Cancelling IRPs



I'm trying to figure out how to cancel an IRP waiting for a controller object but I am finding it a lot more difficult then I had imagined.


Cancelling IRPs Waiting For Controller Objects

This technical tip started out a lot more attractive, but as with most things involving cancel routines in Windows NT kernel mode drivers, it got a little ugly. The approach presented here is viable, but its complexity would most likely make it appropriate only in circumstances that demanded it. In laying out its solution, however, this tip does discuss some interesting aspects of controller objects and cancel routines.

Controller objects can be powerful tools when designing Windows NT kernel mode drivers. They offer the ability to synchronize driver resources between different device objects within the driver. To this end, each device object can queue a request on the controller object using the function IoAllocateController. When the controller becomes available to service the needs of that device, a callback routine will be invoked. At this point, the device that receives control owns the resource and can perform its work. If this can be done within the context of the callback routine, the device can return DeallocateObject, allowing the controller to go on to service other requests, or it can return KeepObject, and at some later time call the function IoFreeController.

Typically, there may be several devices in a driver that each contend for the resources managed by the controller object. For example, a piece of hardware may contain two separate subsystems, but have only one interface that must be shared between them for control. Architecturally, it may make sense to model each subsystem as its own device. The shared interface could be managed with a controller object, allowing the two devices to exist independently of one another.

It is standard practice for any IRP that is placed in a pending state on a device, to provide a cancel routine and make the IRP cancelable prior to entering the pending state. The most common example of this occurs when an IRP must be serialized, and is placed on a device queue to await its turn to be processed on the device. By placing the IRP into a cancelable state, the driver's responsiveness and flexibility are enhanced. If the issuer of the IRP determines that the IRP is no longer necessary, for instance the process that issued the IRP has been terminated, the IRP can be quickly disposed of on the device. In the case of a device queue, the IRP can be removed from the queue if it hasn't started on the device yet, and completed in a canceled state. In the above example of a terminating process, once all of the outstanding IRPs have been completed or canceled, the file handle to the device can be closed, and the process termination can continue to completion.

Waiting on a controller object to process an IRP is in a sense a pending state, for which one may consider trying to employ a cancel routine and making the IRP cancelable while waiting for the controller access to be granted. Of course this is not required, and the driver could take the approach that once a request has been queued to the controller object, it has been started on the device and cannot be canceled. This could pose a problem, however, if some device holds onto the controller object for a relatively long time, and outstanding IRPs from other devices that have queued requests to the controller cannot be canceled.

When attempting to make an IRP that is waiting on a controller object cancelable, there are several potential pitfalls that must be avoided. To understand what they are, one must first understand what happens when a request is queued on a controller object. Performing this action is done using the function IoAllocateController, which has the prototype:





IN PDRIVER_CONTROL ExecutionRoutine,

IN PVOID Context


Notice that the function takes a pointer to the device object, which is queued on the controller object. What this actually means is important to understand. Within the call to IoAllocateController, the system takes the KDEVICE_QUEUE_ENTRY WaitQueueEntry member, located in the WAIT_CONTEXT_BLOCK structure, which is part of the DEVICE_OBJECT structure (see ntddk.h), and queues it to the KDEVICE_QUEUE DeviceWaitQueue member located in the controller's CONTROLLER_OBJECT structure.

An important point to note is that the DEVICE_OBJECT only contains one such entry, meaning that it can only be placed on the list once, otherwise the list will become corrupted. Furthermore, a device object can only be queued on a single list at any given time. Also note that this same technique is employed in the operation of DMA_ADAPTER objects, which make use of the same list entry in the device object.

Because of the singular list entry, device objects must serialize their attempts to queue themselves on controller (or DMA adapter) objects. Thus, it is important in any attempt to cancel an IRP that is pending on an outstanding queued request, to try and remove the request from the controller's queue. This must be done atomically with respect to the system, which will also try to remove the request from the queue for processing on the device. This can be done using the function KeRemoveEntryDeviceQueue, which has the prototype:






For the parameter DeviceQueue, one would use


and for DeviceQueueEntry, one would use


If the device object cannot be successfully removed from the controller's device queue, the IRP cannot be safely canceled in the cancel routine, since starting the next IRP on the device may try to queue the device again on the controller object, thus corrupting the list. Actually, most sane design methodologies would indicate that an outstanding request on a device must be fully resolved, and the device state known, before the IRP can be completed or canceled. This ensures that there will be no problems in either starting the next IRP, or with extraneous processing occurring after the cancelled IRP has been completed, which can be undesirable.

The following is pseudo-code for the steps that must be taken in the various routines to make an IRP that is pending on a controller object cancelable. The main details are presented, but there are several "routine" details that are omitted for clarity sake. The DriverWorks™ controller example CONTROLR has been updated using the same logic presented here. These fragments assume that the IRP has simply been serialized prior to attempting to gain control of the resources managed by the controller, and that the device state has not been altered by the IRP in any other way that needs to be corrected in the cancel routine.

In StartIo routine:

Remove the IRP from the cancelable state it was in while queued, testing to see if the IRP was canceled, and dealing with it appropriately if it was.

Enter QueueSync spinlock, a private spinlock created for the sole purpose of allowing the cancel routine to be set and the request queued to the controller atomically, with no chance that the cancel routine will proceed before the request is queued.

Place the IRP into a cancelable state, using the CancelInProgress routine as its cancel routine, while testing to see if the IRP was canceled, and dealing with it appropriately if it was.

Queue our request on the controller object.

Leave the QueueSync spinlock.

In CancelInProgress routine:

Release global cancel spinlock.

Enter QueueSync spinlock.

Leave QueueSync spinlock. Now it is assured that the request has been queued.

Try to remove the device's KDEVICE_QUEUE_ENTRY from the controller's KDEVICE_QUEUE.

If successful, complete the IRP in the canceled state, and start the next IRP on the device.

If not successful, signal an event, CancelComplete. This will indicate to the callback routine that the cancel routine has finished executing. Note in this case, the IRP is not completed, nor the next IRP started from the cancel routine.

In ControllerCallback routine:

Remove IRP from the cancelable state it was in while queued on the controller object, testing to see if the IRP has been canceled.

If the IRP has been canceled, queue a work item WaitForCancel and deallocate the controller object.

If the IRP has not been canceled, proceed with the work that the device must do to satisfy the IRP, keeping the controller object until that work is done and then releasing it.

In WaitForCancel routine:

Wait on the CancelComplete event.

Complete the IRP in the canceled state, and start the next IRP on the device.

There are several interesting points to discuss concerning the above logic. Most of these points involve tricky timing issues that come to light when analyzing the task of synchronizing two asynchronous threads of execution, namely the cancel thread and the controller callback thread. This endeavor leads to most of the complexity shown in the pseudocode.

First, consider the spinlock QueueSync, which is used in the StartIo and CancelInProgress routines. This spinlock is necessary, since without it the cancel routine could try to dequeue the device object from the controller's queue before it has been queued, in which case it will not find the request. This would result in queuing a canceled request to the controller. Ultimately, this situation would be resolved when the request is dequeued by the system, but it will not happen in a timely manner, which is the intent in the first place. One very important point to note, with regards to this spinlock, is that the ControllerCallback routine may be called with the spinlock held if the system finds the controller is not busy and calls the routine inline to the queuing request on the controller. This is not a problem so long as the callback routine does not try to reacquire the spinlock, which would result in a deadlock.

Next, consider an IRP that is canceled and the possible ordering of events. The IRP may be canceled while the request is still queued on the controller object. In this case the cancel routine will be able to successfully remove it from the queue, complete it as canceled, and start the next IRP on the device.

Another possibility is that the system dequeues the request and calls the ControllerCallback routine before the IRP is canceled. If the IRP is canceled after the callback routine succeeds in making the IRP not cancelable, the system will not call the cancel routine, and the IRP will complete normally when processing is completed on the device. Note that it is not compulsory to complete an IRP that has been canceled with a canceled status. Instead, that IRP should simply be completed soon.

A final possibility is that the system dequeues the request just as the cancel routine is being run. In this case, both the cancel routine and the callback routine will execute. When the cancel routine runs, it will not find the request queued on the controller object. Instead of completing the IRP and starting the next IRP, the cancel routine will signal the CancelComplete event indicating it has run. When the callback routine runs, it will detect that the IRP has been canceled, and it will queue a work item to wait for the CancelComplete event. The reason a work item must be used is that the callback routine may be running at DEVICE IRQL, precluding the ability to wait. The reason to wait on the event in the WaitForCancel routine is to ensure that both the cancel and callback processing have been completed, before completing this IRP or starting the next IRP on the device. This ensures that our device state is fully resolved before the next IRP is started.

The ability to cancel pending requests on controller objects may not be required for most driver designs. Certainly, any design in which requests will not remain pending for very long before being serviced will not benefit greatly from the added complexity. In a small subset of designs, however, this ability might be critical. It is somewhat unfortunate, therefore, that the underlying mechanics are somewhat hidden in the call to the IoAllocateController routine, since it forces our solution to rely on that mechanism remaining constant. While it is unlikely that this mechanism will change, the fact remains that its design makes use of the driver's device object in a semi-opaque manner, instead of opening up the interface by exposing the underlying device queue mechanism. Indeed, the design could have been made much more generalized by the inclusion of a parameter allowing the KDEVICE_QUEUE_ENTRY to be specified in the call to IoAllocateController, eliminating the dependence on the singular instance enmeshed in the driver's device object.

Old KB# 11311
Comment List
Related Discussions