Troubleshooting batch jobs in AX

DAXRunBase

7 years ago

If your batch jobs are seemingly not executing in Microsoft Dynamics AX, there could be a multitude of reasons. This guide will help with troubleshooting batch jobs in AX by listing the possible causes and providing solutions for each points.

Initial setup

It is important to define what is a batch job first:

A batch job is a process which will carry out specific tasks at a certain date and time running as a background process within the AX Application Object Server service.

We need to tell the system what is the process that we want execute, which consists of a Batch job as a header, and at least one Batch task which is the actual processes running. The batch jobs can be grouped together by their roles with the help of a Batch group. If none is provided, it would still classify as one called the “Empty batch group”.

In typical AX Production installations you would want to isolate your batch execution from the rest of the business, so you would dedicate an AX AOS instance to be the batch server. Under System administration module > Setup > System > Server configuration make sure you set the following:

Is batch server checkbox enabled for your dedicated batch server
Batch server schedule has minimum 1 active thread, and the Start/End time has the default value of 00:00:00 – 23:59:59 to allow batch execution any time during the day. I typically set up 2 threads per physical CPU core available (16 for the recommended 8-core AOS for a Prod instance)
Batch server groups must have at least the Empty batch group listed, but if you associate batch jobs with other batch groups, they must be also allocated to one of the active batch servers.

A more detailed explanation and generic setup can be found in the online MSDN documentation with great detail, I would recommend to read through all the sub-pages to get a good understand and to validate your initial setup:

https://technet.microsoft.com/EN-US/library/gg731793.aspx

With a fresh AX deployment you have two batch jobs created out of the box, you can read up a bit about them and also with possible issues associated with them in the following article:

https://kaya-consulting.com/standard-batch-jobs-ax2012/

Stable environment

The most important prerequisite for successfully running batch jobs is to have a stable environment. In majority of the cases the biggest cause that batch jobs keeps sitting in a Waiting status is that the application is not stable.

Validate the following points to make sure your environment is stable:

You have a fully compiled Application Object Tree (AOT), that has 0 errors for all nodes. You must compile the AOT at least one time, after which you may use the axbuild tool for consequent, faster compilations
After you are sure that your system is error-free, you need to Generate Full CIL without errors
Restart the AOS instance after the above steps to make sure the assembly and the netmodule DLL files are correctly deployed
Once the AOS instance is back, verify if the Standard AX AIF ports do not have any stop signs under System administration module > Setup > Services and Application Integration Framework > Inbound ports.

Debugging

Once your environment is considered to be stable, the next problem is your Technical staff (developers). Leaving active soft or hard breakpoints (text in X++) in the system in combination with attaching to the AX AOS service process with Visual Studio will stop the execution of all threads for the server, including batch jobs.

Solution:

Disable Breakpoints for X++ code running on the server, Global breakpoints and hot-swapping of assemblies in Windows Control Panel > Administrative tools > Microsoft Dynamics AX Server Configuration Utility
Connect to the SQL Server instance that hosts the AX database, and remove breakpoints by running the following SQL statement against the data DB:
```
truncate table SysBreakpoints;
GO
truncate table SysBreakpointList;
GO
```
Restart the AX AOS service dedicated for batch execution to ensure there are no active debugging sessions going on

Execution account

Batch jobs are running under the AX AOS service account for the outside world, so if you are for example accessing files or folders in a network share, you have to make sure the AOS execution account has appropriate security permissions.

If someone is using a Development environment, it is common to run the AOS service under the developer’s active directory account. They usually forget to change the password also on the AOS service and restart it once the company policy forces them to set a new password every couple of months, which can break different parts of the system including batch job execution

For processes within the server thread they are impersonating the AX User who has scheduled the batch job through Windows Integrated Security. On Production instances the companies should be using one or more dedicated service accounts for batch execution. Selecting Right click > Run as a different user on the AX shortcut or client executable would allow you to open the AX client as a different account to schedule the batch jobs.

In many cases the user or batch accounts are locked out due to mistyping passwords (if the domain admin forces password lockout policy), in which case your system engineers should re-enable them in the Active Directory. A locked or deleted AD account is also a common cause why batch jobs are not running.

One more important setting is to associate the correct Security Roles for the batch execution accounts (System User is a must, and in many cases System Administrator are being used), and the account must be Enabled.

64-bit assembly

Batch jobs are executing within the Microsoft Dynamics AX Application Object Server service, which is a 64-bit process. In case you would like to call methods which require a client-bound process (such as trying to display a report on screen, open a form, ask for user input), or a 32-bit DLL/assembly, that is not going to work.

You can read up more about the problem and a possible solution here:

https://blog.mattmags.com/2007/06/30/accessing-32-bit-dlls-from-64-bit-code/

Threads

Within the AX Server configuration, you could define that during what time period how many batch threads could operate parallel in each AOS services.

If multiple batch jobs or tasks are scheduled to run at the same time, it is possible that the maximum number of threads are already exhausted, making addition Waiting batch tasks to be idle, until another thread becomes available. One example is the MRP scheduling with multiple helpers, which is essentially just splitting the batch job to multiple smaller parallel tasks, each consuming an available thread.

If everything else fails for troubleshooting batch jobs

The ultimate tip for troubleshooting batch jobs is to set up Visual Studio for AX debugging, attach to the Ax32serv.exe process, and place breakpoint on \Classes\BatchRun\serverGet*Task methods, which are being called by the kernel process for fetching the next available batch jobs to be executed. Stepping through the code should allow you to see what is missing.