[Development tutorial] Thoughts on test automation + more sample tests using ATL

I recently had an interesting discussion around testing and specifically Microsoft testing efforts and frameworks, namely, ATL. If you are not familiar with this framework - you've been living under a rock. Get out from under there and go read some articles about it, like the one below

https://kashperuk.blogspot.com/2019/05/development-tutorial-sample-test-tips.html

The discussion revolved around the complexity and costs of adding and maintaining tests. The partner also questioned how often we add tests, and if we can share more samples of such. You can also see his comment with a more concrete ask in the blog post above. 

Here's a partial quote:

Partners need more example for this functionality usage. Is it possible that you provide real tests for some real issues (I just took several bugs from the latest PU Warehouse and Transportation Management)

533573 Creating a sales order’s work without a pick location is not updating the on-hand figures correctly.

535311 Cannot do material consumption from mobile device for shelf life items

536899 The container contents report does not show an error if shipment does not have a dropoff address specified

537772 Cluster profile could not be found please check your configuration Error is blocking Purchase Order Receiving

If you share them in a new blog post as examples that will be a great discussion point

So in this post, I'd like to share 2 things:

1. Generally speaking, we at Microsoft, or rather, more specifically, we in the Warehouse management team, add one or more tests for every single customer reported issue. 

This obviously adds a cost to each bug fix we do, but with this we get a much more reliable release schedule and on-going confidence in our product offering, and have very few regressions despite heavy code churn.

The point I want to make here is - if you are an ISV, and you have not covered your solution with test automation, you should do it now, and reap the benefits. 

If you are a partner getting paid by the hour doing a specific customer implementation, it gets more tricky. However, I believe (I know) it is in your and your customer's interest to justify the cost of adding test automation.

Very frequently today, even with some of our larger customers, we hear test cycles of 1-2 months before taking a newer standard release. That puts a significant on-going cost on the customer, as well as stress, as we at Microsoft keep pushing everyone to be current. This could have been mostly alleviated with test automation

ATL is a framework which allows to write robust (and reasonably fast) test automation with a low cost investment, all things considered. 

2. We do not ship our test automation library. The code is not written with shipping in mind, so it's not necessarily "Microsoft production quality". The value for customers is also marginal. 

But, to answer the ask from the quote above directly, here are the tests added for the 4 issues listed. I did not include the class/test setup logic, nor did I verify these tests are able to run on your specific dataset (internally we have tests that are both dataset dependent and independent). So take them for what they are - mere samples that prove the point - all of these are rather easy to understand when reading, and are rather easy to write, if you know the scenario you want to automate. For all or most of the standard functionality there are already ATL "wrappers" ready to be used.

• 533573 Creating a sales order’s work without a pick location is not updating the on-hand figures correctly.

	[SysTestCheckInTest]
    public void soRelease_StopWorkOnLocDirFailureFalse_onHandCorrectWorkCompleted()
    {
		// Given
        const InventQty QtyOnReceiptLocation = 100;
        const InventQty SalesQuantity = 1;

        ttsbegin;

        var locDirFailure = whs.locationDirectiveFailures().sales();
        locDirFailure.LocDirFailWork = NoYes::No;
        locDirFailure.update();

        var locProfile = whs.locationProfiles().receipt();
        locProfile.setLPControlled(false).save();

        var receiptLocation = whs.locations(warehouse).create(locProfile);

        // Delete existing sales pick location directive to ensure no impact to the one created.
        var locDir = AtlEntityWHSLocationDirective::find('24 SO Pick', WHSWorkType::Pick, WHSWorkTransType::Sales, warehouse);
        locDir.delete();

        whs.locationDirectives().salesPick().setLocationRange(pickLocation).moveToTop();

        onhand.adjust().forItem(item).forInventDims([receiptLocation]).addOnHandQty(QtyOnReceiptLocation);

        salesOrder.addLine().setItem(item).setInventDims([warehouse]).setQuantity(SalesQuantity).setAutoReservation().save();
        ttscommit;

		// When
        salesOrder.releaseToWarehouse();

		// Then
        var work = salesOrder.work().singleEntity();

        work.lines().withTypePick().withLocationId('').assertSingle();
		
        invent.trans().query().forItem(item).forReferenceCategory(InventTransType::WHSWork)
			.forReferenceId(work.parmWorkId()).assertCount(0);

		onHand.assertExpectedOnHand(
            onHand.spec().forItem(item).withInventDims([warehouse]).withAvailTotal(QtyOnReceiptLocation - SalesQuantity));

        var soPicking = rf.login().salesOrderPicking()
			.setWork(work)
            .setLocation(receiptLocation);

        invent.trans().query().forItem(item).forReferenceCategory(InventTransType::WHSWork)
            .forReferenceId(work.parmWorkId()).withStatusIssue(StatusIssue::ReservPhysical).assertSingle();
        invent.trans().query().forItem(item).forReferenceCategory(InventTransType::WHSWork)
            .forReferenceId(work.parmWorkId()).withStatusReceipt(StatusReceipt::Ordered).assertSingle();

        soPicking
			.confirmPick()
			.confirmPut()
			.assertWorkCompleted();
    }  
  

• 535311 Cannot do material consumption from mobile device for shelf life items

      [SysTestCheckInTest,
        SysTestCaseAutomaticNumberSequences,
        SysTestFeatureDependency(classStr(WHSProdMaterialConsumptionJournalBOMBatchExpiryDateFeature), true)]
    public void materialConsumption_BatchIsExpiredBySchedDate_JournalLinesCreated()
    {
        const Qty BomLineQty = 1;
        const Qty ProductionOrderQty = 1;

        ttsbegin;

        // Given

        InventModelGroup fefoGroup = invent.modelGroups().fefoWithExpiryCriteriaBuilder().save().record();
        fefoGroup.PickingListBatchExpirationDateValidationRule = WHSPickingListBatchExpirationDateValidation::ReservationDate;
        fefoGroup.update();

        InventTable rawMaterial = items.whsBatchAboveBuilder()
            .setItemModelGroup(fefoGroup)
            .setPdsShelfLife(PdsShelfLife)
            .create();

        InventBatch validBatch = invent.batches().create(rawMaterial);
        validBatch.setExpiryDate(DateTimeUtil::getToday(DateTimeUtil::getUserPreferredTimeZone()) + PdsShelfLife).save();

        onhand.adjust().forItem(rawMaterial).forInventDims([floor, lp1, validBatch]).addOnHandQty(RMItemQty);

        InventTable finishedGood = items.whsBatchBelowBuilder()
            .setItemModelGroup(fefoGroup)
            .setPdsShelfLife(PdsShelfLife)
            .create();

        AtlEntityBillOfMaterials billOfMaterials = this.createBOMAndBOMVersion(rawMaterial, BomLineQty, finishedGood);
        AtlEntityProductionOrder productionOrder = this.createProductionOrder(finishedGood, billOfMaterials, ProdReservation::None, ProductionOrderQty);
        productionOrder.estimate().execute();
        
        productionOrder.scheduleOperation().setSchedDirection(ProdSchedDirection::ForwardFromSchedDate).setFiniteCapacity(false)
            .setFiniteMaterial(false).setSchedDate(DateTimeUtil::getToday(DateTimeUtil::getUserPreferredTimeZone()) + PdsShelfLife + PdsShelfLife).execute();

        productionOrder.start().setAutoBOMConsumption(BOMAutoConsump::Never)
            .setPostPickingList(false).execute();

        ttscommit;

        AtlWhsWorkExecuteProdMaterialConsumption materialConsumption = this.createMaterialConsumptionWithBatch(productionOrder, floor, rawMaterial, validBatch);
        
        materialConsumption.setQuantityToConsume(ProductionOrderQty);

        // When and Then
        materialConsumption.assertMessageOnClickOk("@WAX:MaterialConsumptionJournalLineCreated");

        materialConsumption.assertMessageOnDone(
            strFmt(
                "@WAX:MaterialConsumptionJournalPosted",
                productionOrder.pickingLists().withJournalType(ProdJournalType::Picklist).singleEntity().parmJournalId()));
    }
  

• 536899 The container contents report does not show an error if shipment does not have a dropoff address specified (Not using ATL, as this was an old existing test. More than 1 test was added / modified here)

      [SysTestCheckInTest]
    public void preRunValidate()
    {
        // arrange
        InventSiteLogisticsLocation inventSiteLogisticsLocation;
        delete_from inventSiteLogisticsLocation;

        Args args = new Args(formStr(WHSContainerTable));
        FormRun formRun = classfactory.formRunClass(args);
        formRun.init();

        FormDataSource containerTableDS = formRun.dataSource(tableStr(WHSContainerTable));
        containerTableDS.executeQuery();
        containerTableDS.markRecord(containerTable, true);

        args = new Args();
        args.caller(formRun);
        args.record(containerTable);

        WHSContainerContentsControllerTestable controller = new WHSContainerContentsControllerTestable();
        controller.parmArgs(args);

        Query query = new Query(queryStr(WHSContainerContents));
        controller.setRanges(query);

        Map queryContract = new Map(Types::String,Types::Class);
        queryContract.insert('Query',query);

        SrsReportDataContract srsReportDataContract = new SrsReportDataContract();
        srsReportDataContract.parmQueryContracts(queryContract);
        controller.parmReportContract(srsReportDataContract);

        // act
        container validateResults = controller.preRunValidate();

        // assert
        this.assertEquals(validateResults, [SrsReportPreRunState::Error, "@WAX:WHSContainerContentsReportMissingPrimaryAddress"], 'Unexpected value in validateResults.');
    }
  

• 537772 Cluster profile could not be found please check your configuration Error is blocking Purchase Order Receiving

      [SysTestCheckInTest]
    public void poReceive_QualityOrderAndAssignClusterEnabled_WorkAssignedToCluster()
    {
        const Qty PurchOrderQuantity = 2;
        ttsbegin;
        // Given
        whs.warehouses().ensureQualityManagementEnabled(warehouse);
        invent.qualityOrders().ensureCanCreate();
        whs.locationDirectives().qualityItemSampling(whs.locations(warehouse).stage().wMSLocationId, warehouse);
        whs.workTemplates().qualityItemSampling();
        invent.qualityAssociations().defaultBuilder()
            .setOrderType(InventTestReferenceType::Purch)
            .setApplicableWarehouseType(WHSApplicableWarehouseType::QualityManagementOnlyEnabled)
            .setItemRelation(whsItem.ItemId)
            .setSiteId(warehouse.InventSiteId)
            .setDocumentType(InventTestDocumentStatus::Registration)
            .setShowInfo(NoYes::Yes)
            .setAcceptableQualityLevel(100)
            .setQualityProcessingPolicy(WHSQualityProcessingPolicy::CreateQualityOrder)
            .setItemSampling(invent.itemSamplings().onePiece())
            .getResult();
        clusterProfile = whs.clusterProfiles()
            .putawayClusterManualAssignAtReceipt(WHSWorkTransType::Purch, whs.workTemplates().purchaseReceipt().parmWorkTemplateCode());
        ttscommit;
            
        this.addPurchaseOrderLine(whsItem, PurchOrderQuantity);

        // When
        var poItemRecv = rf.login().purchaseOrderItemReceiving()
            .setPurchaseOrder(purchOrder)
            .setItem(whsItem)
            .setQuantity(PurchOrderQuantity);

        AtlQueryWHSWork workQuery = purchOrder.work().withClusterProfileId(clusterProfile.parmProfileId());
                
        // Then
        poItemRecv.assertErrorOnConfirmUnitSelectionWithWorkQuery(workQuery, clusterProfile.parmProfileId(), 'Wrong message');
        invent.qualityOrders().query().forPurchaseOrder(purchOrder.record()).assertCount(1, 'Should have 1 quality order');
    }
  

Hope this helps. Do reach out to us with feedback / suggestions.

Do make use of CDE (Community Driven Engineering) to submit additions / bug fixes. We do expect ATL tests with those fixes.

[Development tutorial] Sample test + tips for using ATL (Acceptance Test Library) to implement tests in Warehouse management

Hopefully, by now everyone is familiar with the fact that we shipped our internal test framework for all ISVs, partners and customers to use.
We have published pretty detailed documentation about the framework on docs.microsoft.com:

https://docs.microsoft.com/en-us/dynamics365/unified-operations/dev-itpro/perf-test/acceptance-test-library

I, however, learn much better when looking at samples. We have shipped a couple sample tests in each area with the 10.0.3 release.
In this post, I would like to share another sample test case and walk through the different types of commands possible in ATL, focusing also on the part that drives mobile device flow operations.

Important note

ATL is not meant as a replacement for good quality production code, and should not be used to execute business logic in PROD environments

Sample test

This test will validate a very simple inbound flow where I want to validate I can confirm the putaway location by the location check digit:

• Create a new purchase order for a WHS-enabled item to a WHS-enabled warehouse
• Perform the registration of the item through the mobile device
• Perform the putaway through the mobile device
• Confirm the location confirmation is shown and is asking for the check digit
• Enter the check digit of the putaway location instead of the location ID
• Complete the putaway
• Validate the work lines are updated to Closed (not really the point of the check digit test, but just to show the capability)

[SysTestMethod]
public void confirmLocationWithCheckDigit_SimplePurchOrderOneItem_Success()
{
 ttsbegin;

 //========Given========

 purchaseOrder.addLine().setItem(item).setInventDims([warehouse]).setQuantity(PurchaseQuantity).save();

 rf.login().purchaseOrderItemReceiving().setPurchaseOrder(purchaseOrder).setItem(item).setQuantity(PurchaseQuantity).confirmUnitSelection();

 var work = purchaseOrder.work().singleEntity();

 //========When========

 var poPutaway = rf.login().purchaseOrderPutaway().setWork(work).confirmPick();

 poPutaway.parmCheckDigitControl().assertIsEnabled(); // Will automatically happen within set method below
 poPutaway.setCheckDigit(checkDigit); // This will confirm the Put to this location

 //========Then========

 poPutaway.assertWorkCompleted();

 work.reread();
 this.assertEquals(WHSWorkStatus::Closed, work.parmWorkStatus(), 'Work should be closed');

 work.lines().assertExpectedLines(
  whs.workLines().spec().withLineNum(1).withItem(item).withStatusClosed().withQuantity(PurchaseQuantity).withLocation(whs.locations().receipt()),
  whs.workLines().spec().withLineNum(2).withItem(item).withStatusClosed().withQuantity(PurchaseQuantity).withLocation(putawayLocation));

 ttsabort;
}

Looks pretty simple, doesn't it? And does not really require comments, as the code is pretty self explanatory for the most part, "even a functional consultant can read it" :)
A number of things I would like to call out here:

• Notice the name of the test method. We always try to name the method according to the Given/When/Then pattern (also explicitly called out in the method body), more specifically, When_Given_Then(). Or, for those more familiar with AAA, Arrange_Act_Assert()
• Note the method name does not start with test. You can still do that, but we prefer being explicit through decorating the method with the SysTestMethod attribute
• I skipped the test setup part - I'll post that below. Normally that would include steps generic for all tests, while the Given part in here would only contains a few things special to this particular test. In this test, we 
• create a purchase order line for a small quantity of a WHS-enabled item
• do the PO receipt through the mobile device, to create the putaway work.
• There is A LOT of defaulting that happens in these fluent calls. When writing code, you need to pay attention and override any default value. For example, 
• When setting inventory dimensions for the PO line, I only specify the warehouse, but that in turn sets the Site as part of the business logic. The Inventory status is also defaulted based on the Warehouse parameters. 
• When logging it to the mobile device in rf.login(), the default worker is used. However you can overwrite that and specify another worker to log in with. 
• The WMA flows should look like you'd have it on the actual device, with scanner actually sending a caret return aka clicking OK after entry. Note, that defaulting is also supported here, where we just proceed without explicitly specifying a unit
• Here, however, I used a dedicated method, confirmUnitSelection(), to improve readability. Inside, in the implementation, it's a simple OK button click. 
• It is possible to override this behavior, by disabling the automatic submission on the flow class. This way, you'll have the ability to for example set multiple controls at the same time before OK'ing, etc. 
• ATL allows for most areas to rely on either a special entity class, or the underlying record. Generally speaking, we always prefer to use the entity over the record. For example, take a look at how I retrieve the work entity for the putaway work created after registration.
• Process Guide flows are tested with a different set of classes at this point. We'd be interested to hear your feedback on, which approach is more convenient. 
• When doing putaway, I also use a "dummy" method, confirmPick() to confirm the pick up of the LP from the RECV location. These methods, however, could be doing some additional validation
• We have, for example, implemented some like confirmPutToBaydoor(), which will confirm we are in fact in a Put step, and the Location control has a value where it is of type "Final shipping location". Think through these scenarios when extending ATL
• We can also access the controls shown on the current screen right now, to perform additional validation, for example. I showcase that with the CheckDigit control, simply checking that it is enabled for data entry.
• This already happens automatically in the next call to setCheckDigit(), so you should consider where you really need this.
• assertIsEnabled() will also ensure the control is in fact visible.
• Lastly, I wanted to show how specs are used in a real example, where we in assertExpectedLines() specify the 2 lines we expect to see on the putaway work after execution of the putaway. This allows for really flexible validation of the data, that also is easy to read and understand.
• Check out how the flow class shows up in the debugger - it displays the XML with the controls, which really helps understand where you are in the flow if something in the test goes very wrong!

Entire test class

Below is the full listing of the xpp class with the test, including variable declarations and setup.

You can also download it from my OneDrive

[SysTestGranularity(SysTestGranularity::Integration),
SysTestCaseAutomaticNumberSequences]
public final class DEV_PurchaseOrderReceivingAndPutawayScenarioTests extends SysTestCase
{
    AtlDataRootNode                     data;
    AtlDataInvent                       invent;
    AtlDataInventOnHand                 onHand;
    AtlDataWHS                          whs;
    AtlDataWHSMobilePortal              rf;
    AtlDataPurch                        purch;

    InventTable                         item;
    InventLocation                      warehouse;
    WMSLocation                         putawayLocation;
    AtlEntityPurchaseOrder              purchaseOrder;

    private const WMSCheckText          CheckDigit = '12345';
    private const PurchQty              PurchaseQuantity = 1;

    public void setUpTestCase()
    {
        super();

        data = AtlDataRootNode::construct();
        invent = data.invent();
        onHand = data.invent().onHand();
        whs = data.whs();
        rf = whs.rf();
        purch = data.purch();

        ttsbegin;

        whs.helpers().setupBasicPrerequisites();
        AtlWHSWorkUserContext::currentWorkUser(whs.workUsers().default());

        item = invent.items().whs();
        warehouse = whs.warehouses().whs();
        whs.warehouses().ensureDefaultReceiptLocation(warehouse);

        putawayLocation = whs.locations().floor();
        putawayLocation.checkText = CheckDigit;
        putawayLocation.update();

        whs.locationDirectives().purchasePutaway();
        whs.workTemplates().purchaseReceipt();

        var poPutawayMenuItem = rf.menuItems().purchaseOrderPutaway();
        rf.menuItems().setupWorkConfirmation(poPutAwayMenuItem.MenuItemName, WHSWorkType::Put, NoYes::No, NoYes::Yes);

        rf.menus().default().addMenuItem(rf.menuItems().purchaseOrderItemReceiving())
                            .addMenuItem(poPutawayMenuItem);

        purchaseOrder = data.purch().purchaseOrders().createDefault();

        ttscommit;
    }

 [SysTestMethod]
 public void confirmLocationWithCheckDigit_SimplePurchOrderOneItem_Success()
 {
  ttsbegin;

  //========Given========

  purchaseOrder.addLine().setItem(item).setInventDims([warehouse]).setQuantity(PurchaseQuantity).save();

  rf.login().purchaseOrderItemReceiving().setPurchaseOrder(purchaseOrder).setItem(item).setQuantity(PurchaseQuantity).confirmUnitSelection();

  var work = purchaseOrder.work().singleEntity();

  //========When========

  var poPutaway = rf.login().purchaseOrderPutaway().setWork(work).confirmPick();

  poPutaway.parmCheckDigitControl().assertIsEnabled(); // Will automatically happen within set method below
  poPutaway.setCheckDigit(checkDigit); // This will confirm the Put to this location

  //========Then========

  poPutaway.assertWorkCompleted();

  work.reread();
  this.assertEquals(WHSWorkStatus::Closed, work.parmWorkStatus(), 'Work should be closed');

  work.lines().assertExpectedLines(
   whs.workLines().spec().withLineNum(1).withItem(item).withStatusClosed().withQuantity(PurchaseQuantity).withLocation(whs.locations().receipt()),
   whs.workLines().spec().withLineNum(2).withItem(item).withStatusClosed().withQuantity(PurchaseQuantity).withLocation(putawayLocation));

  ttsabort;
 }

}

A few extra notes:

• We recommend that you create a base test class for your test automation, that will hide the ATL navigation variable declaration and initialization, along with potentially some common setup, like I have at the beginning of the setupTestCase() method, setupBasicPrerequisites() and worker initialization.
• Note the use of SysTestCaseAutomaticNumberSequences attribute on the class. This allows not to worry about configuring them manually, just let the framework do it for you.
• Note that my test will not use an existing partition and company. It should generally work in USMF as well though, and it's up to you to decide how much data setup you want to rely on.

OK, that's it for this post. It's a bit of a brain dump, but please read through the points, and don't hesitate to reach out with questions.

Hope this helps

Tool: Update: Automated way of setting up the Application Integration Framework

In November 2009 I have posted a blog post that presented a tool I wrote for quickly setting up AIF for testing purposes. You can find the original post here.

I have since received requests to upload a new version of that tool, that would work with AX 2012 R2.
Well, now, only 5 years later :), here is the updated tool, that works with AX 2012 R3 (Note: should also work with R2, but I have only tried it on R3).

You can download the xpo from my OneDrive.

To briefly describe the changes, as compared to the version from AX 2009:

• There is no longer a concept of endpoints in AX 2012. Instead, there is a concept of inbound and outbound ports, so I have replaced the corresponding creation so that 1 inbound and 1 outbound ports are created, both using the File System Adapter, and dropping the files in a location on drive D. Note, that I have not done anything in terms of ensuring AOS has access to that folder, so if that is not working, ensure the right access is granted.
• Channels are now created implicitly for each port, depending on the Adapter class you select.
• I am not explicitly creating data policies now, which seems to mean all XML fields are enabled. If you want to restrict the schema to only certain fields, you will need to deactivate the port, and modify the data policies accordingly manually.

Note: Only services for Invent*, EcoRes* and WHS* prefixes are registered, and only their service actions are added to the ports. If you need additional actions in your tests, modify the code and add the extra prefix.

I have tested that the WHS ASN service works after executing the tool, both for inbound and outbound, and will describe the use of this service in a blog post on the SCM msdn blog shortly. If you however find stuff I have missed, let me know.

Thanks

How to quickly set up Application Integration Framework (AIF) for testing Axd documents

Purpose of this post
Part of the test effort on our team at MDCC goes for verifying the Axd documents shipped with the Dynamics AX application.

As many people find it hard to setup AIF, I have tried to help them by creating an automated way of doing the initial setup, which can be used for testing the documents using a 1-box setup and FileSystem transport adapter.

Download link
From the following link, you will be able to download an X++ class that initializes all the entities required for AIF to work, like the endpoints, transport adapters and channels, services, their actions and action policies.
Note, that since I am working on the Inventory team, I am filtering out the services accordingly. You can easily extend/modify this to suite your specific needs by changing the filter or adding new ones in classDeclaration and method main

Download xpo

Example from the standard application
To give a specific example, I encourage you to try sending a selected item electronically in form of an XML file right after executing the job.
To do that, navigate to Inventory Management\Common forms\Item details and press Send electronically button. Fill in the values for endpoint, filter on a specific item(s), and press OK.

Your screen should look similar to this:

Processing the document
The document is successfully placed into the queue for processing and sending through the gateway. If you have a batch job for processing AIF services up and running, you will soon see a file in the outbound files folder setup on the outbound channel.
If you would like to learn, how to set up the batch jobs for this, follow the below link: Start and stop the asynchronous AIF services

In order to avoid having to set the batch jobs up, I have added 2 extra methods to the class for starting the processing of inbound and outbound messages.

public static void ProcessOutboundDocuments()
{
    new AifOutboundProcessingService().run();
    new AifGatewaySendService().run();
}
public static void ProcessInboundDocuments()
{
    new AifGatewayReceiveService().run();
    new AifInboundProcessingService().run();
}

Just call the DEV_AifInitialSetup::ProcessOutboundDocuments method from a job, and the document will be automatically processed and sent on its way to the outbound documents folder.

Output
The resulting XML file for the item should look similar to the following: