[Development tutorial] Thoughts on test automation + more sample tests using ATL

I recently had an interesting discussion around testing and specifically Microsoft testing efforts and frameworks, namely, ATL. If you are not familiar with this framework - you've been living under a rock. Get out from under there and go read some articles about it, like the one below

https://kashperuk.blogspot.com/2019/05/development-tutorial-sample-test-tips.html

The discussion revolved around the complexity and costs of adding and maintaining tests. The partner also questioned how often we add tests, and if we can share more samples of such. You can also see his comment with a more concrete ask in the blog post above. 

Here's a partial quote:

Partners need more example for this functionality usage. Is it possible that you provide real tests for some real issues (I just took several bugs from the latest PU Warehouse and Transportation Management)

533573 Creating a sales order’s work without a pick location is not updating the on-hand figures correctly.

535311 Cannot do material consumption from mobile device for shelf life items

536899 The container contents report does not show an error if shipment does not have a dropoff address specified

537772 Cluster profile could not be found please check your configuration Error is blocking Purchase Order Receiving

If you share them in a new blog post as examples that will be a great discussion point

So in this post, I'd like to share 2 things:

1. Generally speaking, we at Microsoft, or rather, more specifically, we in the Warehouse management team, add one or more tests for every single customer reported issue. 

This obviously adds a cost to each bug fix we do, but with this we get a much more reliable release schedule and on-going confidence in our product offering, and have very few regressions despite heavy code churn.

The point I want to make here is - if you are an ISV, and you have not covered your solution with test automation, you should do it now, and reap the benefits. 

If you are a partner getting paid by the hour doing a specific customer implementation, it gets more tricky. However, I believe (I know) it is in your and your customer's interest to justify the cost of adding test automation.

Very frequently today, even with some of our larger customers, we hear test cycles of 1-2 months before taking a newer standard release. That puts a significant on-going cost on the customer, as well as stress, as we at Microsoft keep pushing everyone to be current. This could have been mostly alleviated with test automation

ATL is a framework which allows to write robust (and reasonably fast) test automation with a low cost investment, all things considered. 

2. We do not ship our test automation library. The code is not written with shipping in mind, so it's not necessarily "Microsoft production quality". The value for customers is also marginal. 

But, to answer the ask from the quote above directly, here are the tests added for the 4 issues listed. I did not include the class/test setup logic, nor did I verify these tests are able to run on your specific dataset (internally we have tests that are both dataset dependent and independent). So take them for what they are - mere samples that prove the point - all of these are rather easy to understand when reading, and are rather easy to write, if you know the scenario you want to automate. For all or most of the standard functionality there are already ATL "wrappers" ready to be used.

• 533573 Creating a sales order’s work without a pick location is not updating the on-hand figures correctly.

	[SysTestCheckInTest]
    public void soRelease_StopWorkOnLocDirFailureFalse_onHandCorrectWorkCompleted()
    {
		// Given
        const InventQty QtyOnReceiptLocation = 100;
        const InventQty SalesQuantity = 1;

        ttsbegin;

        var locDirFailure = whs.locationDirectiveFailures().sales();
        locDirFailure.LocDirFailWork = NoYes::No;
        locDirFailure.update();

        var locProfile = whs.locationProfiles().receipt();
        locProfile.setLPControlled(false).save();

        var receiptLocation = whs.locations(warehouse).create(locProfile);

        // Delete existing sales pick location directive to ensure no impact to the one created.
        var locDir = AtlEntityWHSLocationDirective::find('24 SO Pick', WHSWorkType::Pick, WHSWorkTransType::Sales, warehouse);
        locDir.delete();

        whs.locationDirectives().salesPick().setLocationRange(pickLocation).moveToTop();

        onhand.adjust().forItem(item).forInventDims([receiptLocation]).addOnHandQty(QtyOnReceiptLocation);

        salesOrder.addLine().setItem(item).setInventDims([warehouse]).setQuantity(SalesQuantity).setAutoReservation().save();
        ttscommit;

		// When
        salesOrder.releaseToWarehouse();

		// Then
        var work = salesOrder.work().singleEntity();

        work.lines().withTypePick().withLocationId('').assertSingle();
		
        invent.trans().query().forItem(item).forReferenceCategory(InventTransType::WHSWork)
			.forReferenceId(work.parmWorkId()).assertCount(0);

		onHand.assertExpectedOnHand(
            onHand.spec().forItem(item).withInventDims([warehouse]).withAvailTotal(QtyOnReceiptLocation - SalesQuantity));

        var soPicking = rf.login().salesOrderPicking()
			.setWork(work)
            .setLocation(receiptLocation);

        invent.trans().query().forItem(item).forReferenceCategory(InventTransType::WHSWork)
            .forReferenceId(work.parmWorkId()).withStatusIssue(StatusIssue::ReservPhysical).assertSingle();
        invent.trans().query().forItem(item).forReferenceCategory(InventTransType::WHSWork)
            .forReferenceId(work.parmWorkId()).withStatusReceipt(StatusReceipt::Ordered).assertSingle();

        soPicking
			.confirmPick()
			.confirmPut()
			.assertWorkCompleted();
    }  
  

• 535311 Cannot do material consumption from mobile device for shelf life items

      [SysTestCheckInTest,
        SysTestCaseAutomaticNumberSequences,
        SysTestFeatureDependency(classStr(WHSProdMaterialConsumptionJournalBOMBatchExpiryDateFeature), true)]
    public void materialConsumption_BatchIsExpiredBySchedDate_JournalLinesCreated()
    {
        const Qty BomLineQty = 1;
        const Qty ProductionOrderQty = 1;

        ttsbegin;

        // Given

        InventModelGroup fefoGroup = invent.modelGroups().fefoWithExpiryCriteriaBuilder().save().record();
        fefoGroup.PickingListBatchExpirationDateValidationRule = WHSPickingListBatchExpirationDateValidation::ReservationDate;
        fefoGroup.update();

        InventTable rawMaterial = items.whsBatchAboveBuilder()
            .setItemModelGroup(fefoGroup)
            .setPdsShelfLife(PdsShelfLife)
            .create();

        InventBatch validBatch = invent.batches().create(rawMaterial);
        validBatch.setExpiryDate(DateTimeUtil::getToday(DateTimeUtil::getUserPreferredTimeZone()) + PdsShelfLife).save();

        onhand.adjust().forItem(rawMaterial).forInventDims([floor, lp1, validBatch]).addOnHandQty(RMItemQty);

        InventTable finishedGood = items.whsBatchBelowBuilder()
            .setItemModelGroup(fefoGroup)
            .setPdsShelfLife(PdsShelfLife)
            .create();

        AtlEntityBillOfMaterials billOfMaterials = this.createBOMAndBOMVersion(rawMaterial, BomLineQty, finishedGood);
        AtlEntityProductionOrder productionOrder = this.createProductionOrder(finishedGood, billOfMaterials, ProdReservation::None, ProductionOrderQty);
        productionOrder.estimate().execute();
        
        productionOrder.scheduleOperation().setSchedDirection(ProdSchedDirection::ForwardFromSchedDate).setFiniteCapacity(false)
            .setFiniteMaterial(false).setSchedDate(DateTimeUtil::getToday(DateTimeUtil::getUserPreferredTimeZone()) + PdsShelfLife + PdsShelfLife).execute();

        productionOrder.start().setAutoBOMConsumption(BOMAutoConsump::Never)
            .setPostPickingList(false).execute();

        ttscommit;

        AtlWhsWorkExecuteProdMaterialConsumption materialConsumption = this.createMaterialConsumptionWithBatch(productionOrder, floor, rawMaterial, validBatch);
        
        materialConsumption.setQuantityToConsume(ProductionOrderQty);

        // When and Then
        materialConsumption.assertMessageOnClickOk("@WAX:MaterialConsumptionJournalLineCreated");

        materialConsumption.assertMessageOnDone(
            strFmt(
                "@WAX:MaterialConsumptionJournalPosted",
                productionOrder.pickingLists().withJournalType(ProdJournalType::Picklist).singleEntity().parmJournalId()));
    }
  

• 536899 The container contents report does not show an error if shipment does not have a dropoff address specified (Not using ATL, as this was an old existing test. More than 1 test was added / modified here)

      [SysTestCheckInTest]
    public void preRunValidate()
    {
        // arrange
        InventSiteLogisticsLocation inventSiteLogisticsLocation;
        delete_from inventSiteLogisticsLocation;

        Args args = new Args(formStr(WHSContainerTable));
        FormRun formRun = classfactory.formRunClass(args);
        formRun.init();

        FormDataSource containerTableDS = formRun.dataSource(tableStr(WHSContainerTable));
        containerTableDS.executeQuery();
        containerTableDS.markRecord(containerTable, true);

        args = new Args();
        args.caller(formRun);
        args.record(containerTable);

        WHSContainerContentsControllerTestable controller = new WHSContainerContentsControllerTestable();
        controller.parmArgs(args);

        Query query = new Query(queryStr(WHSContainerContents));
        controller.setRanges(query);

        Map queryContract = new Map(Types::String,Types::Class);
        queryContract.insert('Query',query);

        SrsReportDataContract srsReportDataContract = new SrsReportDataContract();
        srsReportDataContract.parmQueryContracts(queryContract);
        controller.parmReportContract(srsReportDataContract);

        // act
        container validateResults = controller.preRunValidate();

        // assert
        this.assertEquals(validateResults, [SrsReportPreRunState::Error, "@WAX:WHSContainerContentsReportMissingPrimaryAddress"], 'Unexpected value in validateResults.');
    }
  

• 537772 Cluster profile could not be found please check your configuration Error is blocking Purchase Order Receiving

      [SysTestCheckInTest]
    public void poReceive_QualityOrderAndAssignClusterEnabled_WorkAssignedToCluster()
    {
        const Qty PurchOrderQuantity = 2;
        ttsbegin;
        // Given
        whs.warehouses().ensureQualityManagementEnabled(warehouse);
        invent.qualityOrders().ensureCanCreate();
        whs.locationDirectives().qualityItemSampling(whs.locations(warehouse).stage().wMSLocationId, warehouse);
        whs.workTemplates().qualityItemSampling();
        invent.qualityAssociations().defaultBuilder()
            .setOrderType(InventTestReferenceType::Purch)
            .setApplicableWarehouseType(WHSApplicableWarehouseType::QualityManagementOnlyEnabled)
            .setItemRelation(whsItem.ItemId)
            .setSiteId(warehouse.InventSiteId)
            .setDocumentType(InventTestDocumentStatus::Registration)
            .setShowInfo(NoYes::Yes)
            .setAcceptableQualityLevel(100)
            .setQualityProcessingPolicy(WHSQualityProcessingPolicy::CreateQualityOrder)
            .setItemSampling(invent.itemSamplings().onePiece())
            .getResult();
        clusterProfile = whs.clusterProfiles()
            .putawayClusterManualAssignAtReceipt(WHSWorkTransType::Purch, whs.workTemplates().purchaseReceipt().parmWorkTemplateCode());
        ttscommit;
            
        this.addPurchaseOrderLine(whsItem, PurchOrderQuantity);

        // When
        var poItemRecv = rf.login().purchaseOrderItemReceiving()
            .setPurchaseOrder(purchOrder)
            .setItem(whsItem)
            .setQuantity(PurchOrderQuantity);

        AtlQueryWHSWork workQuery = purchOrder.work().withClusterProfileId(clusterProfile.parmProfileId());
                
        // Then
        poItemRecv.assertErrorOnConfirmUnitSelectionWithWorkQuery(workQuery, clusterProfile.parmProfileId(), 'Wrong message');
        invent.qualityOrders().query().forPurchaseOrder(purchOrder.record()).assertCount(1, 'Should have 1 quality order');
    }
  

Hope this helps. Do reach out to us with feedback / suggestions.

Do make use of CDE (Community Driven Engineering) to submit additions / bug fixes. We do expect ATL tests with those fixes.

[Development tutorial] Sample test + tips for using ATL (Acceptance Test Library) to implement tests in Warehouse management

Hopefully, by now everyone is familiar with the fact that we shipped our internal test framework for all ISVs, partners and customers to use.
We have published pretty detailed documentation about the framework on docs.microsoft.com:

https://docs.microsoft.com/en-us/dynamics365/unified-operations/dev-itpro/perf-test/acceptance-test-library

I, however, learn much better when looking at samples. We have shipped a couple sample tests in each area with the 10.0.3 release.
In this post, I would like to share another sample test case and walk through the different types of commands possible in ATL, focusing also on the part that drives mobile device flow operations.

Important note

ATL is not meant as a replacement for good quality production code, and should not be used to execute business logic in PROD environments

Sample test

This test will validate a very simple inbound flow where I want to validate I can confirm the putaway location by the location check digit:

• Create a new purchase order for a WHS-enabled item to a WHS-enabled warehouse
• Perform the registration of the item through the mobile device
• Perform the putaway through the mobile device
• Confirm the location confirmation is shown and is asking for the check digit
• Enter the check digit of the putaway location instead of the location ID
• Complete the putaway
• Validate the work lines are updated to Closed (not really the point of the check digit test, but just to show the capability)

[SysTestMethod]
public void confirmLocationWithCheckDigit_SimplePurchOrderOneItem_Success()
{
 ttsbegin;

 //========Given========

 purchaseOrder.addLine().setItem(item).setInventDims([warehouse]).setQuantity(PurchaseQuantity).save();

 rf.login().purchaseOrderItemReceiving().setPurchaseOrder(purchaseOrder).setItem(item).setQuantity(PurchaseQuantity).confirmUnitSelection();

 var work = purchaseOrder.work().singleEntity();

 //========When========

 var poPutaway = rf.login().purchaseOrderPutaway().setWork(work).confirmPick();

 poPutaway.parmCheckDigitControl().assertIsEnabled(); // Will automatically happen within set method below
 poPutaway.setCheckDigit(checkDigit); // This will confirm the Put to this location

 //========Then========

 poPutaway.assertWorkCompleted();

 work.reread();
 this.assertEquals(WHSWorkStatus::Closed, work.parmWorkStatus(), 'Work should be closed');

 work.lines().assertExpectedLines(
  whs.workLines().spec().withLineNum(1).withItem(item).withStatusClosed().withQuantity(PurchaseQuantity).withLocation(whs.locations().receipt()),
  whs.workLines().spec().withLineNum(2).withItem(item).withStatusClosed().withQuantity(PurchaseQuantity).withLocation(putawayLocation));

 ttsabort;
}

Looks pretty simple, doesn't it? And does not really require comments, as the code is pretty self explanatory for the most part, "even a functional consultant can read it" :)
A number of things I would like to call out here:

• Notice the name of the test method. We always try to name the method according to the Given/When/Then pattern (also explicitly called out in the method body), more specifically, When_Given_Then(). Or, for those more familiar with AAA, Arrange_Act_Assert()
• Note the method name does not start with test. You can still do that, but we prefer being explicit through decorating the method with the SysTestMethod attribute
• I skipped the test setup part - I'll post that below. Normally that would include steps generic for all tests, while the Given part in here would only contains a few things special to this particular test. In this test, we 
• create a purchase order line for a small quantity of a WHS-enabled item
• do the PO receipt through the mobile device, to create the putaway work.
• There is A LOT of defaulting that happens in these fluent calls. When writing code, you need to pay attention and override any default value. For example, 
• When setting inventory dimensions for the PO line, I only specify the warehouse, but that in turn sets the Site as part of the business logic. The Inventory status is also defaulted based on the Warehouse parameters. 
• When logging it to the mobile device in rf.login(), the default worker is used. However you can overwrite that and specify another worker to log in with. 
• The WMA flows should look like you'd have it on the actual device, with scanner actually sending a caret return aka clicking OK after entry. Note, that defaulting is also supported here, where we just proceed without explicitly specifying a unit
• Here, however, I used a dedicated method, confirmUnitSelection(), to improve readability. Inside, in the implementation, it's a simple OK button click. 
• It is possible to override this behavior, by disabling the automatic submission on the flow class. This way, you'll have the ability to for example set multiple controls at the same time before OK'ing, etc. 
• ATL allows for most areas to rely on either a special entity class, or the underlying record. Generally speaking, we always prefer to use the entity over the record. For example, take a look at how I retrieve the work entity for the putaway work created after registration.
• Process Guide flows are tested with a different set of classes at this point. We'd be interested to hear your feedback on, which approach is more convenient. 
• When doing putaway, I also use a "dummy" method, confirmPick() to confirm the pick up of the LP from the RECV location. These methods, however, could be doing some additional validation
• We have, for example, implemented some like confirmPutToBaydoor(), which will confirm we are in fact in a Put step, and the Location control has a value where it is of type "Final shipping location". Think through these scenarios when extending ATL
• We can also access the controls shown on the current screen right now, to perform additional validation, for example. I showcase that with the CheckDigit control, simply checking that it is enabled for data entry.
• This already happens automatically in the next call to setCheckDigit(), so you should consider where you really need this.
• assertIsEnabled() will also ensure the control is in fact visible.
• Lastly, I wanted to show how specs are used in a real example, where we in assertExpectedLines() specify the 2 lines we expect to see on the putaway work after execution of the putaway. This allows for really flexible validation of the data, that also is easy to read and understand.
• Check out how the flow class shows up in the debugger - it displays the XML with the controls, which really helps understand where you are in the flow if something in the test goes very wrong!

Entire test class

Below is the full listing of the xpp class with the test, including variable declarations and setup.

You can also download it from my OneDrive

[SysTestGranularity(SysTestGranularity::Integration),
SysTestCaseAutomaticNumberSequences]
public final class DEV_PurchaseOrderReceivingAndPutawayScenarioTests extends SysTestCase
{
    AtlDataRootNode                     data;
    AtlDataInvent                       invent;
    AtlDataInventOnHand                 onHand;
    AtlDataWHS                          whs;
    AtlDataWHSMobilePortal              rf;
    AtlDataPurch                        purch;

    InventTable                         item;
    InventLocation                      warehouse;
    WMSLocation                         putawayLocation;
    AtlEntityPurchaseOrder              purchaseOrder;

    private const WMSCheckText          CheckDigit = '12345';
    private const PurchQty              PurchaseQuantity = 1;

    public void setUpTestCase()
    {
        super();

        data = AtlDataRootNode::construct();
        invent = data.invent();
        onHand = data.invent().onHand();
        whs = data.whs();
        rf = whs.rf();
        purch = data.purch();

        ttsbegin;

        whs.helpers().setupBasicPrerequisites();
        AtlWHSWorkUserContext::currentWorkUser(whs.workUsers().default());

        item = invent.items().whs();
        warehouse = whs.warehouses().whs();
        whs.warehouses().ensureDefaultReceiptLocation(warehouse);

        putawayLocation = whs.locations().floor();
        putawayLocation.checkText = CheckDigit;
        putawayLocation.update();

        whs.locationDirectives().purchasePutaway();
        whs.workTemplates().purchaseReceipt();

        var poPutawayMenuItem = rf.menuItems().purchaseOrderPutaway();
        rf.menuItems().setupWorkConfirmation(poPutAwayMenuItem.MenuItemName, WHSWorkType::Put, NoYes::No, NoYes::Yes);

        rf.menus().default().addMenuItem(rf.menuItems().purchaseOrderItemReceiving())
                            .addMenuItem(poPutawayMenuItem);

        purchaseOrder = data.purch().purchaseOrders().createDefault();

        ttscommit;
    }

 [SysTestMethod]
 public void confirmLocationWithCheckDigit_SimplePurchOrderOneItem_Success()
 {
  ttsbegin;

  //========Given========

  purchaseOrder.addLine().setItem(item).setInventDims([warehouse]).setQuantity(PurchaseQuantity).save();

  rf.login().purchaseOrderItemReceiving().setPurchaseOrder(purchaseOrder).setItem(item).setQuantity(PurchaseQuantity).confirmUnitSelection();

  var work = purchaseOrder.work().singleEntity();

  //========When========

  var poPutaway = rf.login().purchaseOrderPutaway().setWork(work).confirmPick();

  poPutaway.parmCheckDigitControl().assertIsEnabled(); // Will automatically happen within set method below
  poPutaway.setCheckDigit(checkDigit); // This will confirm the Put to this location

  //========Then========

  poPutaway.assertWorkCompleted();

  work.reread();
  this.assertEquals(WHSWorkStatus::Closed, work.parmWorkStatus(), 'Work should be closed');

  work.lines().assertExpectedLines(
   whs.workLines().spec().withLineNum(1).withItem(item).withStatusClosed().withQuantity(PurchaseQuantity).withLocation(whs.locations().receipt()),
   whs.workLines().spec().withLineNum(2).withItem(item).withStatusClosed().withQuantity(PurchaseQuantity).withLocation(putawayLocation));

  ttsabort;
 }

}

A few extra notes:

• We recommend that you create a base test class for your test automation, that will hide the ATL navigation variable declaration and initialization, along with potentially some common setup, like I have at the beginning of the setupTestCase() method, setupBasicPrerequisites() and worker initialization.
• Note the use of SysTestCaseAutomaticNumberSequences attribute on the class. This allows not to worry about configuring them manually, just let the framework do it for you.
• Note that my test will not use an existing partition and company. It should generally work in USMF as well though, and it's up to you to decide how much data setup you want to rely on.

OK, that's it for this post. It's a bit of a brain dump, but please read through the points, and don't hesitate to reach out with questions.

Hope this helps