This page collects several stated problems and candidate solutions. This is a place for discussion and reflection that will probably take a different form in the future.
The technical detail should not be the main focus in this page, although proposed solutions have to take them into account.
Currently, the core provides special Operations, such as Apparition to represent the effect of inventory processes, yet AnyBlok / Wms Base doesn’t include a representation of inventories, that should in turn be the major source of these Operations.
Depending on the needs, there are several possible ways to proceed with inventories, but they usually amount to record what is actually present in the relevant locations, compare that with the database and issue corrections.
We should provide some support for that process, but it would be in a new optional Blok, in order not to block applications that would need to proceed in a very different way.
It’s been a design requirement from the beginning that Avatars of a given physical object don’t overlap, but it’s hard to enforce and gives rise to Avatars whose lifespan is too fictive, meaning that it’s not a prediction made by applicative code or the end user, but just minimal garbage so that the Avatars don’t overlap. It goes as far as having many Avatars with a zero length lifespan.
An issue not adressed yet is the one of execution of chains of planned Operations (#8 on Github). Namely, once the first Operation gets executed, it sets the real dates and times on its outcomes. No effort is currently made to adjust downstream Avatars to avoid their overlaping with the ones now being present, and that happens in case the actual time of execution is later than planned (almost always the case with the default value of the planned time of execution). It could be solved by propagating any positive shift, but that defeats one of our core goals: that execution of planned Operations should be really fast.
It would be better to use None
as a marker for totally unkwnown
times of execution and datetimes of future
Avatars. Besides to be
more understandable than having weird values in the database, it would
lift the burden of propagating a shift in the most common use case
where there was simply no planned date of execution (we should enforce that
any downstream Avatar of an indeterminate one should be, too),
but this gives rise to a new set of problems:
semantically, dt_until == None
currently means the end of times.
If we use instead to to just say
that we don’t know, we need something else to mark that an
Avatar is terminal in the current state of planification. Maybe
that it’s not the input of any Operation is good enough, maybe we’d
have to mark it explicitely with a boolean for efficiency reasons,
maybe we’d end up chaining Avatars directly.
in future
quantity queries, we’d need to avoid counting Avatars of a
given physical object twice, and we’d prefer the results to stay additive
with respect to the hierarchy of location/containers, ie, if location
A contains exactly two sublocations B and C:
quantity(location=A, …) = quantity(location=B, …) + quantity(location=C…)
Otherwise, it could lead to subtle bugs in applicative code that would implicitely take that additivity for granted, or mere confusion and distrust of end users that’d browse the stock levels.
Here we should be really careful to treat (and test!) all corner
cases, such as when
dt_until
is None but dt_from
isn’t and is earlier than the
required datetime etc.
would we have Avatars with both dt_from
and dt_until
being
None
or would we rather propagate dt_from
to take into
account that, at least, we known that execution will happen no
matter what after a certain date ? If both can be None
we’d have
some weird effects: a planned Arrival with
a given date, followed by a Move would lead to the outcome of the
Move been counted in future
quantity queries before the planned
time of the Arrival, so we better prevent that to happen. On the other hand,
does this not repeat the problem with shifting the whole chain at
execution of the first Operation ? Perhaps this would be less
problematic, as the double countings due to overlapping Avatars
would still be solved ? After all, the future
quantity queries that
are the most important are the ones at infinity (not affected by this),
and we could mitigate with an asynchronous shifting (at least we
wouldn’t have double countings in the meanwhile).
Finally, if we could go as far as to allow overlapping Avatars, we could imagine to support simultaneous concurrent scenarios in planification, and maybe implementations of Superseding of planned operations in a way that’d be akin to the obsolescence markers of Mercurial, but that’s probably too far fetched.
Decision about all this postponed until version 0.9
In a big system, especially with several sites for Goods handling (warehouses, retail stores), the detail of operations occurring at some given premises is usually of no interest for the big picture.
For example, we could have a central system taking care of sales and purchases, and keeping track of rough stock levels for these purposes.
Such a system would certainly not be interested by the detailed organization of locations inside the different warehouses, nor with the many operations that occur as part of the reception, keep in stock, then delivery process and in fact, it would burden it. On the other hand, it’s best if handling sites don’t suffer the network latency to an offsite system.
The central system could instead have a simplified view of the logistics, representing each handling site as a single Location, maybe using Goods lines with quantities whereas a handling site would not, and intercommunication would happen over the bus or REST APIs that are planned anyway for Anyblok WMS.
If well done, that can also play some kind of sharding role, but there are intrinsic limits as to how much simplified the view of the central system can be, even combined with Superseding of planned operations to transmit only simplified operations.
Note
about the central system example
For mass scalability, keeping an exact track of stock levels is irrealistic anyway: the logistics system is too big and has too much processing to do to ask it for realtime reports.
At a certain scale, its reports would timeout or fall out of sync because of, actually, general failure under the stress they generate. All the federation system can achieve in that case is pushing back the point of failure.
Besides, if one managed 100 orders per minute, how useful is it to track them by the unit to tell customers if they are available ?
Obviously, many different scenarios can be achieved with well-thought federation, including mesh-like moving of Goods across sites, as needed if one has several production sites and several retail stores.
Communication with other systems also fall in this category.
Well, yeah, this page should be superseded. How ?
New in version 0.9.0: we now have enough planning alteration primitives to support the use case detailed here, but we don’t have the most general form of Operations graph manipulation suggested near the end of this section.
We should provide the means to declare that a chain of operations actually replaces (and maybe completes) some given (chain of?) planned operations.
It’s a general good practice for applications to try and not predict the future too precisely, because of the “stubbornness of reality” but it can lead to dilemmas.
Here’s a concrete case, which I stumbled onto two days ago for one of the prime applications to be built on Anyblok WMS / Base.
The context:
In general, I would advise against representing those incoming parcels and the intermediate boxes if possible, but :
My customer tells me out of other experiences that this is all fairly common in many businesses, and I’m inclined to believe him about that.
Note, at this point, WMS Base does not include anything for verification of unpacks and arrivals, nor any reservation or planning system (that would issue chains of planned Operations), but we have to take into account that end applications will need and have some.
Currently, here is how we can attempt to represent this use case with what the core provides us (none of these is satisfactory):
Under-representation scenario
Drawback: we have a WMS system that doesn’t track some items that are physically carried over in the warehouse! What happens, e.g, if one of the parcels has to be temporarily kept in another location than the normal unpacking area due to some unforeseen condition ?
Over-representation scenario
Let’s not even speak about the intermediate boxes.
Drawbacks:
No crystal ball scenario
This has the obvious merit of being simple, and may be suitable for protoyping, while better alternative are developed.
Drawbacks:
The proposal is that we could merge scenarios 1 and 2 if we’d allow to substitute a planned operation with a chain of operations.
id=1
) in the unpacking area, linked with the
Purchase Ordercontents
storing the theoretical contents, and
link them to the Purchase Orderid=1
), and it would
start planning the Moves (id=5,6,7
) of the parcels to the unpacking
area, as well as their Unpack operations (id=8,9,10
)id=1
) with the chain made of ids 2
through 10, since the contents are identical. The core would arrange
for the unpack outcomes (still unplanned, but that doesn’t matter)
to actually be the already existing incomes of the downstream
operations, which don’t need to be cancelled. Reservations don’t
have to be updated due to the Arrivals being different than id=1
.id=1
had been
executed directly.This proposal doesn’t say anything about which commits or savepoints
are issued to the database and their logical orderings: these can be
considered implementation details at this point, all that matters at
this functional level is that the outcomes of the final Unpacks
with id=8,9,10
id=1
)As already noted, this does not take into account the fact that we’d probably get a single delivery order for those three parcels, but that can be addressed separately by introducing a multi-unpack operation (details of that don’t belong here).
I’m pretty much convinced that the ability to refine a prediction with another one (possibly partly done, it doesn’t matter) would be a great feature, and a strong step towards coping with the stubbornness of reality.
Actually, about any planning would benefit from such a core feature. The motto for downstream developers would then be: “plan the minimum, refine it later to adjust to reality”.
Question: do other WMS have such future history rewrite capabilities?
I’m not sure how far it should go in the general form. Mathematically, it would be about replacing any subgraph of the history DAG by another one which has the same incomes and outcomes, for a suitable definition of “same”.
Maybe it’s simpler to implement it in full generality rather than some special cases like the example above, in which the subgraph has a single root with no incomes, that happens to be also root in the whole DAG.
New in version 0.8.0.
In some cases, one wants to put the goods into some containing object, dsand then perhaps move that containing object. The use cases I have currently are cables in a plastic box and audio devices in a flight case. Let’s use the first one as example.
Currently, if one considers the box as a Location, this leaves the cables it holds accessible to perform operations on them : perhaps move them out of the box, test them and mark them as working or not, etc. But, it does not represent the very convenient thing that can happen in the physical reality: close the lid, move the whole at once into a truck.
On the other hand, one can choose to represent the box as a Goods record, and
load them via an Assembly operation. Then its contents
property
will have the Goods that are stored in the box, but each time one
wants to use or test a cable, one has to perform an Unpack and an
Assembly again. One would have to ignore that the Unpack will produce
avatar
for all the cables in the Location where the box sits, hence much confusion:
in reality, the cables are still in the box, not aside of it.
Moreover, unless special effort is done to avoid that, each
pack/unpack cycle would lead to change of ids, meaning that the system
considers that the box has changed enough to be a new box.
On top of that, the contents are not visible in quantity queries…
Add the issues mentioned in “Location” terminology is misleading on top of that, and it’s clear we have a design problem to solve.
In real life, the plastic box is both an object that can be tossed into a truck and that can hold other objects, so why should we do thing differently in an application meant to represent physical objects ?
We could :
location
field of Avatars point towards a Goods recordcontents
propery
(variable part of Unpack) and with packing/unpacking in
general.Assuming this doesn’t introduce unsolvable problems, this would also take care of all the issues of “Location” terminology is misleading:
parent
of the existing hierarchy, we have
the standard Avatar location
field to indicate that some
location is inside another: it’s now very clear
that it’s about the position in space of the location, instead of
maybe some logical grouping.This would leave us mostly with two concepts: Goods (physical objects) and Operations, which is probably intellectually satisfying, but we’d have a new problem: “Goods” now would sound too specific and would have to be replaced by a more general name (Item ? Object ? PhysObj ?)
New in version 0.8.0: (actually made obsolete by Droping Locations altogether in favor of Goods)
Our concept of Location does not imply that it is actually a fixed place. Locations can actually be moving ones (a van, a ship, a trolley or even a carrying box if needed).
I’ve heard that some proprietary WMS system makes use of the word “support” for the same purposes. It sounds a bit obscure to my taste, though. What alternatives would we have ?
Similarly, the hierarchy of locations does not mean that they are actually inside each other. It’s rather some kind of logical grouping, useful to aggregate stock levels, or to confine some Goods to a group of Locations once they are reserved.
New in version 0.7.0.
Some applications will have many of Goods Types, which will be often mere variations of each other, for example clothes of different sizes.
It is therefore natural to group them in one way or another, both for direct consideration by applicative code, and to allow mutualisation of configuration within WMS Base.
Namely, we could make the Physical Object Types Model hierarchical, by
means of a parent
field. This would bring the following
possibilities:
If a behaviour is not found on a given
Goods Type, then it would be looked up recursively on its parent,
meaning that direct access to the behaviours
field in applicative code
should be prohibited, in favour of the get_behaviour()
method,
that would take care of the inheritance.
We could also allow merging of behaviours: a Goods Type could
for instance inherit the unpack
behaviour from its parent,
changing only the required_properties
key/value pair. Nested
mapping merging is rather simple. Merging lists would be more
complicated to specify.
In some cases, it’d be interesting to specify an intermediate node in the Physical Object Types hierarchy rather than the most precise one. This could be useful for instance in Assembly Operations.
Help resolve the hard choices between Physical Object Types and Properties by providing a way to convert the Type of some Goods to a more precise one according to its Properties.
The interesting thing is that this could be done without any
change in the id
of the Goods, which is how we decided to
represent that the physical object itelf is unchanged: only our way
to consider has actually changed.
This has the drawback that a given Goods record could be represented in several ways, and that is definitely not sane. Some logic, such as quantity queries, could be aware of it at a high complexity price. Perhaps the good way to do it would be to make it transparent:
set_property()
method
set the proper Goods Type automatically on changes of that
Property. (Not done for 0.7.0)get_property()
method
return the proper value for that Property, inferred from the
actual Goods Type. (This is actually a consequence of the Type
Properties, also done for 0.7.0)This transparency would also simplify configuration of Assembly Operations when faced with generic types: no need for even more complex configuration to decide on the final Goods Type, just treat it like any other Property.
Also, it would help refactoring applications that would start by considering a parameter to be a simple Property, and later on recognize that they need to represent it as a full Goods Type.
Note
replaced by simple location type filterings in version 0.8.0
New in version 0.7.0.
Counting (or summing) the goods quantities is expensive within an arborescent structure, even if done with PostgreSQL recursive queries.
And actually, it’s often a bad idea to rely on the arborescence for that. Imagine a system with two warehouses: it’s tempting to have a location for each warehouse, that would be the ancestor of all locations within the warehouse. Now do we really like to count all items in there, including locations for temporary storage of damaged goods before actually destroying them ?
In fact, measuring stock levels is often done for a purpose (like deciding whether we can sell), and, assuming we want an exact count, it should not rely on the Location hierarchy, but rather on the Location’s purpose (e.g., storage before shipping to customers) or not on Locations at all.
Therefore we should introduce a simple tag system for stock levels grouping in Location. Getting back to the crucial example of counting sellable goods, it should also take any notion of reservation into account anyway (it’s truer than counting short term previsions).
We can keep the arborescent structure, claiming this time that it really expresses physical inclusion of Locations (can be useful for other purposes than stock levels, such as confinement of reserved Goods). It could be acceptable that by default, these tags are copied to the sub-Locations upon creation, but nothing more.
We should rename the parent
field as part_of
or inside
to
insist on that.
New in version 0.7.0.
Note
at the time of this writing, The PhysObj Model: Physical Objects had the
quantity
field that is now carried by
wms-quantity.
In the current state of the project, The PhysObj Model: Physical Objects records have a
quantity
field. There are several hints that this shouldn’t be a part
of the core, but should be moved to a distinct blok. Let’s call it
wms-aggregated-goods
for the time being.
we settled on Decimal
(Python) / numeric
(PostgreSQL) to
account for use cases resorting to physical measurements (lengths of
wire, tons of sand). Of course that’s overridable, but it’s an
example of the core taking decisions it should not
this creates a non trivial complexity for most operations, that have to maybe split Goods records.
in most logistics applications, only packaged Goods are actually been handled anyway, therefore they are merely equivalent to units (reels of 100m of wiring, bags of 50kg sand, etc.).
The obvious and only benefits of this quantity
field in these use cases
are that we can represent many identical such units with a single
line in the database.
But these benefits are severely impaired by the need to perform and record many Splits, unless it’s so much common to handle several of them together and not as some kinds of bigger packs, such as pallets or containers that it counterbalances the overhead of all those Splits.
Thery are also impaired by traceability requirements, for instance
if the related properties have consequent variability. In the extreme
case, if we track serial numbers for all goods, then we’ll end up
with each Goods record having quantity=1
.
In many use cases, including the most prominent one at the inception of WMS Base, several identical goods almost never get shipped to final customers, so it’s guaranteed that the overwhelming majority of these lines of Goods with quantities greater that 1 would be split down to quantity 1, and even if we’d defined the Unpacks outcomes to have single Goods lines with quantity equal to 1, it would still not be the worth carrying around the code that decides whether to split or not.
On the other hand, putting aside the current code for quantities and the related operations would probably create a rift in the implementations.
Namely, wms-aggregated-goods
would have to override much of
wms-core
and I fear that it’d become under-used, which would
either impair its compatibility with downstream libraries and
applications, or become a needless development burden on these latter ones.
New in version 0.6.0.
Note
at the time of this writing, PhysObj was called “Goods”, there was a separate Model for locations, and Goods bore all the fields that are now in Avatars
Due to the planning and historical features we want, in our system, the physical goods will give rise to many different records of Goods as non destructive operations, typically Moves currently create new records, and obsolete the ones they got as input.
This is a problem to design a reservation system, which should clearly not reserve some Goods in some precise state at some time in some place, but only be attached to the mostly immutable part of their data.
For an example of the latter requirement, consider a T-shirt been reserved in advance for some end delivery, before it has even arrived in the warehouse. Imagine some planner has decided to put it in location AB/X/234 before packing it with other goods of the same delivery and shipping them to the final customer. Now, deciding at the last minute to put it in the adjacent AB/X/235 should not void the reservation. It should require at most partial replanning. Even if the end location is the planned one, but the Goods record isn’t the same one, the system should not have to update its reservations to match it: that’s an obvious source of conflicts, it’s bad for performance, it contradicts many of our Design Goals and, frankly speaking, it’s absurd: everybody would agree it’s « the same T-shirt ».
Simply arranging for Move to create a new record in the
past
state, changing just location, times and state on the moved
one wouldn’t be a solution, as it would require the even
heavier update of all past history. And having Move mutate all the
The PhysObj Model: Physical Objects in place as we intended before realizing we could
provide History leveraging is not doable because of planning…
So, the proposal is to introduce a new Model, Goods Avatar, that would bear the (very) mutable part of the current Goods. This is what Operations would manipulate and reference.
Now the Goods Model would express the otherwise not so much well-defined idea of a physical object that stays “the same”. We should even provide transforming Operations to resolve the question whether some given change (like engraving a personalised message on a watch) means it becomes a different object or not, as it’s after all only a matter of perception that we can’t decide in WMS Base.
The future reservation system(s) would then lock and/or refer to this skimmed down in the Goods Model. In end applications, concrete schedulers/planners would also refer to them, and look for Avatars to create their planned Operations.
This also probably means that the purposes of the separate Properties Model would boild down to deduplication (probably still very much useful).
All of this is made utterly complicated by the issue of quantities, that’s why this proposal mostly doesn’t speak of them, assuming that other problem is solved.