The problem with Scrum

If you were asked this simple question

— Do you expect to be doing Scrum in 20 years from now?

your answer would necessarily be no. Because you know how much the world, and in particular the industry, changes in time.
You are quite sure that, if you were doing Scrum in 2045, something would be very wrong: i) either the world had frozen until then, or ii) you would be completely at odds with reality.
Now lets go for the fairly logical assumption that you will not be operating a psychosis 20 years from now. Its straightforward.
Therefore, you admit that you would be at odds with reality in 2045.
This leads you to the conclusion that you will surely want to move away from Scrum somewhere between today and 2045.
So you have to admit that, between today and 2045, you will be merely postponing the inevitable until you do away with Scrum.
Therefore, not only do you not have a reason to keep doing Scrum, but also the reason that made you do something else by 2045 is not affected by the timing. Besides this, Scrum was created more than 20 years ago.
Therefore, you arrive at the logical conclusion that, even if you stopped doing Scrum today, you would already be late.
Take from this that there is today no reason at all why anyone would be managing teams with Scrum.

The heading image of this article (from this source) is an illustration of how I see the daily Scrum ceremonies. For some surely remarkable reason, never have I heard a single highly qualified engineer question the following. The daily stand-up is performed, they believe, because it is important, that is, because the team does something in those ceremonies that is important enough for them to do it everyday as clockwork. Now if this was the case, then one should also admit that this very important thing, whatever it is, would be performed many times a day, because it is important, that is, surely not just one single time each day for just about 15 minutes. We don’t do important things just 15 minutes a day at a pre-set time, we do them whenever circumstance justifies so, precisely because they are more important than the other things, so they take the time of the other things when they need to happen. Surely that importance is incompatible with delaying an important matter discussion until the next day huddle. But if we assume this is the case, we should ask why then do we need a daily stand-up?
We have to accept there is some other reason that distinguishes this Scrum ceremony from the supposedly same things that are done during the workday. To understand what that is, we need to look at the specifics of the interaction in the daily stand-up that distinguish it from the other interactions in which the same activities are performed. Firstly, in the daily stand-up all the team is to be present. Second, in Scrum (but unlike the more evolved Disciplined Agile), the team lead is a necessary participant. Can you guess now what is the single thing that justifies the daily stand-up? Its status reporting. Its control. And to this they call “agile”.
This “agile” is more waterfall than Waterfall itself, because in Waterfall we only prescribe daily control over a project during critical periods. Otherwise the timing for the heartbeat is weekly. And to this they call “agile”.
In Lean management you can understand better the abomination of this practice. In Lean, practices follow their own cadences and they happen when they need to instead of merely because the calendar says so. We do not block the flow of work and value with a rock in the middle of the stream every 24 yards.
But as I said, never have I heard a single highly qualified engineer raise this issue, much less solve it. This is of course, because empirical control processes are designed to suppress conflicts about the governing variables. A form of functional lobotomy. And to this they call “agile”.

My inquiries into the industry “agile” standards are grey and old, and that is why you can also read about it in this article.

The rest of this article is based on Al Shalloway’s approach to what is now called Flow for Enterprise Transformation, the state-of-the-art knowledge system that his consulting company developed during the most part of the Agile experiment since its inception, and that was integrated by PMI in its approach for Agile management at the value stream level called DA FLEX. I had three certifications in this method and that is where I come from in this regard. I took the freedom to change very little of Al’s words because they are so perfectly stated, while also adding a few of my own.

Let’s say you were going to buy a car. You walked into the showroom and saw amazing cars and were really excited. You buy the car, have a lot of troubles driving it and when you take it back they respond with:

— Yes, the instructions are simple, but driving the car is difficult to master.
— The car is so sophisticated, it’s not possible to have full predictability.
— Well, we thought this aspect of the car was good enough.
— Most people don’t have trouble with this, but since you do, we have this extra education for you.

How would you react to that? These four attitudes are prevalent in many Agile approaches. If you are a practitioner and wouldn’t accept this for a car I suggest not accepting it for the approach you’re taking to improvement. If you are a consultant, please stop doing it and please help others stop.
Saying something is difficult to master is insidious in that it gives those promoting it permission not to try to get better.
Instead of resign ourselves to a lack of predictability, we should see how much we can get and always get feedback. Looking at a complex problem in a different way often results in a solution that is predictably good. But we’ll never find it if we don’t look for it.
Good enough and too much are not the only alternatives; the use of minimum business increments (MBIs) proves it.
When people don’t understand something about Agile (e.g. tying the big picture to the small task) perhaps it’s not an indication of the person’s inability as much as it’s an indication that something is missing. Scrum has a different education and framework definition approach than Lean, Kanban, Theory of constraints and Flow have, and these are built in DA FLEX but not in Scrum.

The deadly synergy of the four attitudes above results in the following:

i) When someone doesn’t understand something, this attitude will make it easier to just think it’s good enough and just try to explain it better instead of trying to improve it.
ii) When people have difficulty with an approach it’s easier to just shrug our shoulders and explain that it’s difficult to master.
iii) While we can accept that predictability is difficult we should not accept less predictability than we can because our understanding is good enough.

The focus must always be on improving methods and not on following a framework.
The goal is not to be agile at the team level but to achieve business agility across the organization. This manifests both individual and organizational purposes and creates a better place to work. Optimal teams without improvements to the organization is not that useful.

According to the Scrum guide, Scrum is founded on empirical process control theory, or empiricism. Surprisingly, no one seems to challenge that empiricism may not be the best way to base a framework on. On the other hand, DA FLEX is based on the scientific method to overcome the inherent possible cognitive biases. The scientific method is a combination of empiricism and rationalism. It uses empiricism to support or invalidate hypothesis while using rationalism to enable it to create conjectures (i.e., new hypotheses) about how the world works. This combination is more powerful than empiricism alone while being less risky than relying just on academic reasoning.
This mirrors the following paraphrase

Theory without experience is useless, experience without theory is expensive.

Edwards Deming

Several Agile methods, particularly Scrum and most of its derivatives, are based on empiricism, which, while simple to explain, miss many opportunities for a deeper understanding of what is happening. This is not to say we cannot apply the scientific method to Scrum, but doing so requires examining its immutable roles, events, artifacts and rules, possibly leading us to move away from Scrum.

There are a few significant implications of a framework taking on a scientific approach to itself. These include:

i) everything we put forth in the framework is a hypothesis
ii) instead of defending our methods, we welcome, even request, dissent
iii) there cannot be any immutable aspects to the framework

The same way one does not need a complex approach to address a complex problem, as posited by Scrum, one also does not need an empirical approach to address an empirical problem.
DA FLEX is based on the scientific method of making hypotheses, observations and adjusting the hypotheses to fit the data. But no belief or practice is taken as sacrosanct. And deductive logic is encouraged while still requiring the logic to be validated by experience. In addition to this, while Agile was designed and intended to be applied to teams only, DA FLEX takes a systems-thinking point of view which is needed to address organization-wide transformation.

Scrum is a framework. While you are encouraged to add to the framework, certain practices of the framework are sacrosanct – and you are not supposed to change them. As with all frameworks this is to enable a well-defined starting point, however starting with a set framework with core practices that you are not supposed to change tends to have you stick with them longer than you should. Also, not all of Scrum’s practices are applicable everywhere.
While frameworks allow you to add things to them, you often need to change the framework itself. Not all of the prescribed practices in Scrum, which one must do in order to be “doing Scrum” are optimal, or even possible, in all situations.
When a team is being led by someone with only a Scrum certification, they may be reluctant to go away from Scrum because it has them go into territory in which they are not certified and may feel uncomfortable going outside of what they are certified in.

It is difficult to get a man to understand something, when his salary depends upon his not understanding it.

Upton Sinclair

When and how to transcend Scrum is left unanswered by the Scrum Guide which repeatedly encourages following Scrum practices. As the empirical process that it is, Scrum is a particular case of the more general “single loop learning” approaches as PMI termed them. These processes are designed to achieve intended outcomes and suppress conflicts about the governing variables. This means that, while you can make adjustments based on the results you get, you are not allowed to inquire about the method itself, therefore you are left managing a project with the same intelligence as a thermostat manages temperature. You cannot develop a new method even if one would lead you quicker to a solution to the problem you are facing.

We cannot solve our problems with the same thinking we used when we created them.

Albert Einstein

That lack of options to transcend the framework is justified by the seemingly logical statement that “we need a well-defined set of practices; adding options will cause confusion.”

But this is based on the unproven assumption that options have necessarily to be confusing. This is especially relevant to note in the cases when we are managing knowledge workers, because then it equates to a disenfranchisement of the most important assets of an organization — its people doing the work and creating value. You do surely not fear giving options to highly qualified people doing their jobs. In fact, you should fear the opposite, especially if you are paying their wages or waiting for them to solve your problem.
The answer to the claim above is that giving options is not the problem; the challenge is knowing when to give the option. Too soon and people may not have the opportunity to exploit their current approach, but too late means they are stuck with the same method for longer than they need while other alternatives may be better suited for the challenges at hand.
This becomes ever more impactful the more a team develops maturity in Scrum practices. A simple starting point when it needs to be is good, but we have to enable better learning as people get more advanced.
The key after the start is to provide alternatives now that the people know what the objectives of the Scrum practices are. This requires that the people adopting the practices have been given the objective of the practice. At this point, with a little experience under their belt, a deeper set of options can be provided. The key here is not to abandon practices, but to find better ways of doing what you are doing, getting to the root cause of the issue or take on a new practice if needed.
One doesn’t have to blindly follow Scrum but can use Lean-Thinking to provide better, more cost-effective practices to accomplish the intentions of Scrum.
Regarding navigating complexity, Goldratt and the Theory of Constrains identified the term “inherent simplicity”:

If we dive deep enough we’ll find that there are very few elements at the base – the root causes – which through cause-and-effect connections are governing the whole system.

Eli Goldratt

Scrum is also based on the premise that if you change your organization to follow the Scrum framework, Scrum theory, and Scrum values, then your organization will improve. We believe instead that the right approach is to make a decision as to when it’s more effective to change the organization to fit a framework or when to follow different practices that meet the same objectives of the framework more effectively. Of course, this may take you out of the Scrum framework, so you’d no longer be doing Scrum. However, we don’t think this matters. We should remember that we should focus on our objective and not worry about the particular method of getting there.

Struggling with breaking down stories into small chunks? Scrum intentionally provides no guidance here – it is a framework. Unfortunately, standard decomposition methods don’t take advantage of what’s been learned in the last decade with Behavior Driven Development (BDD). BDD provides a structure for decomposition that enables virtually any problem domain to be decomposed into small stories.

If we see people consistently make the same mistakes we should not attribute the failure to the people, but more to the system. This was established 80 years ago by Deming when he stated that the eco-system people find themselves in causes most of the challenges they experience. Our responsibility therefore is to improve the system, more than managing the people.
This is the anti-thesis of what we hear from many Scrum consultants: “Scrum would work if people would just do it.”

That may be true, but Scrum may not be the best approach for the problem a team faces. This approach retains the problem while modifying Scrum to make it invisible so that the dysfunction is no longer a thorn in the side of the team. There is an implication that if the team would put more effort into it, or take the time to figure Scrum out better, we could avoid these. This is a big, assumption.
Our attitude shift would be to ask “why is it so hard to fix this impediment? Is there a better approach, even if it means going beyond Scrum?”

In particular, consultants and promoters of frameworks must take responsibility when practitioners have trouble. This means to instead keep improving our offerings so our clients have fewer challenges.

Neither the Agile Manifesto nor the Scrum Guide mention the role of management once. The role of management is critical, however. It is ironic that Agile is about changing culture but either ignores or vilifies those in the best position to do it.

To contrast with all of the above, we can say the following regarding DA FLEX.

DA FLEX takes the attitude that the overwhelming majority of people are good and, when put in a good system, will achieve good results.
DA FLEX embraces the mindsets of Theory of Constraints, Lean and Flow, seeing no inconsistency between these. While these may be focused more on what we do, DA FLEX also incorporates models of personal and organizational development, which go well beyond the Agile mindset.
DA FLEX is not based on how to take Agile and scale it across the organization. It starts with a holistic view of the organization to begin with.
DA FLEX takes a pragmatic approach, meaning it focuses on what works, which stems from its scientific nature.

Another source for consideration is impartial and unrelated with the PMI. The BPM CBOK v3.0 [1] cites the concept of “Big Process” by Forrester, as a transformation by means of business processes that changes the organization’s focus from isolated BPM actions and incremental improvement initiatives, towards a transformation program that includes the entire organization with the support of executive leadership. This is the vision of the value stream. It is the performance of cross-functional processes, and not of functional areas or assets, that drives the achievement of true results. The broader the BPM initiative in the organization, the more effective it will be in delivering value. This is what DA FLEX is about.

Now for how long will you still be doing Scrum?

_________

[1] the Business Process Management Common Body of Knowledge, by the Association of Business Process Management Professionals

Meister Eckhart’s Sermons

The following are quotes from Meister Eckhart’s Sermons.

1. We may perhaps venture to predict that the Christian biologist of the future will turn the Pauline Christology into his own dialect somewhat after the following fashion:
–“The function of religion in the human race is closely analogous to, if not identical with, that of instinct in the lower animals. Religion is the racial will to live; not, however, to live anyhow and at all costs, but to live as human beings, conforming as far as possible to the highest type of humanity. Religion, therefore, acts as a higher instinct, inhibiting all self-destroying and race-destroying impulses in the interest of a larger self than the individual life.”
To turn this statement into theological form it is only necessary to claim that the “perfect man” which the religious instinct is trying to form is “the measure of the stature of the fulness of Christ,” that perfect humanity was once realised in the historical Christ, and that the higher instinct within us, which makes for life and righteousness, and is the source of all the good that we can think, say, or do, may (in virtue of that historical incarnation) be justly called the indwelling Christ.

2. We must be in absolutely constant relation or mental touch with that essence of life which permeates all and which we call God.
This is almost unrecognisable unless we live into it ourselves actually — that is, by a constant turning to the very innermost, deepest consciousness of our real selves or of God in us, for illumination from within, just as we turn to the sun for light, warmth, and invigoration without.

3. Spiritual supremacy and illumination thus realised through the development and perfection of intuition under divine inspiration gives the perfect inner vision and direct insight into the character, properties, and purpose of all things to which the attention and interest are directed. It is, we repeat, a spiritual sense opening inwardly, as the physical senses open outwardly; and because it has the capacity to perceive, grasp, and know the truth at first hand, independent of all external sources of information, we call it intuition.

4. St Augustine says: the best thing that man can say about God is to be able to be silent about Him, from the wisdom of his inner judgement. Therefore be silent and prate not about God, for whenever thou dost prate about God, thou liest, and committest sin. Thou canst understand nought about God, for He is above all understanding.

5. The masters say: That is young, which is near its beginning.

6. The man who has submitted his will and purposes entirely to God, carries God with him in all his works and in all circumstances. Therein can no man hinder him, for he neither aims at nor enjoys anything else, save God. God is united with Him in all his purposes and designs. Even as no manifoldness can dissipate God, so nothing can dissipate such a man, or destroy his unity.

7. But if the man were wise and diligent, the opposition of the devil and his exercises would be much more profitable to him than the aid of the good angel; for if there were no struggle, there could be no victory.

8. Knock also at the door through which we must go–namely, Christ Jesus.
At this door, the praying man must knock for three ends, if he wishes to be really admitted.
First he must knock devoutly, at the broken heart and the open side, and enter in with all devotion, and in recognition of his unfathomable poverty and nothingness, as poor Lazarus did at the rich man’s gate, and ask for crumbs of His grace.
Then again, he should knock at the door of the holy open wounds of His holy hands, and pray for true Divine knowledge, that it may enlighten him and exalt him.
Finally, knock at the door of His holy feet, and pray for true Divine love, which may unite thee with Him, and immerse and cover thee in Him.

9. Hence a certain doctor says: God is a circle, whose centre is everywhere, and His circumference nowhere.

10. The more simple any being is in itself, the more manifold is it in its energy and operation.

11. Reason mounts up in its own light and by its own power, till at last it comes to think itself the true eternal light, and gives itself out to be such; and it is thus deceived in itself, and deceives others at the same time, people who know no better and are prone to be so deceived.

12. The most noble and gracious gift that is bestowed on any creature is the Reason and the Will. These two are so intimately connected that the one cannot be anywhere without the other. If it were not for these two gifts, there would be no reasonable creatures, but only brutes and brutality; and this would be a great loss, for God would then never receive His due, or behold Himself and His attributes exhibited in action; a thing which ought to be, and is, necessary to perfection.
For it pertains to the will, to will something. For what else does it exist? It would be a vain thing if it had no work to do, and this it cannot have without the creature. And so there must needs be creatures, and God will have them, in order that by their means the will may be exercised, and may work, though in God it must be without work. Therefore the will in the creature, which we call the created will, is as truly God’s as the eternal will, and is not from the creature.

Leonardo da Vinci

Portrait of a man in red chalk, believed to be Leonardo’s self-portrait

The following are statements attributed to Leonardo in his written works [1]. Each paragraph is a different statement from a potentially different original source.

1. But since we know that painting embraces and contains within itself all things produced by nature or whatever results from man’s passing actions – and ultimately everything that can be taken in by the eyes (…) he seems to me to be a pitiful master who can only do one thing well.

2. The good painter has to paint two principal things, that is to say, man and the intention of his mind. The first is easy and the second difficult, because the latter has to be represented through gestures and movements of the limbs.

3. I shall not refrain from including among these precepts a new aid to contemplation, which, although seemingly trivial and almost ridiculous, is none the less of great utility in arousing the mind to various inventions. And this is, if you look at any walls soiled with a variety of stains, or stones with variegated patterns (…), you will therein be able to see a resemblance to various landscapes graced with mountains, rivers, rocks, trees, plains, great valleys and hills in many combinations. Or again you will be able to see various battles and figures darting about, strange-looking faces and costumes, and an endless number of things that you can distil into finely rendered forms. And what happens with regard to such walls and variegated stones is just as with the sound of bells, in whose peal you will find any name or word you care to imagine.

4. I say that when you are painting you ought to have by you a flat mirror in which you should often look at your work. The work will appear to you in reverse and will seem to be by the hand of another master and thereby you will better judge its faults.

5. How to learn well by heart: When you wish to be able to make use of something committed to memory adopt this method, which is that when you have drawn the same thing so many times that it seems you have it by heart try to do it without the exemplar. Have your exemplar traced on to a thin flat plane of glass. Place this on top of the drawing you have done without the exemplar, and note carefully where the tracing does not match up with your drawing, and in those places where you have made a mistake, resolve not to repeat the error. In fact, go back to the exemplar and draw over and over it the erroneous part till you have it firmly in your memory.

6. I say that in narrative paintings you should closely intermingle direct opposites, because they offer a great contrast to each other, and the more so the more they are adjacent. Thus, have the ugly one next to the beautiful, the large next to the small, the old next to the young, the strong next to the weak. In this way there is as much variety, as closely juxtaposed as possible.

7. It previously happened to me that I made a picture representing a holy subject, which was bought by someone who loved it and who wished to remove the attributes of its divinity in order that he might kiss it without guilt. But finally his conscience overcame his sighs and lust, and he was forced to banish it from his house.

8. And if you should have a love for such things you might be prevented by loathing, and if that did not prevent you, you might be deterred by the fear of living in the night hours in the company of those corpses, quartered and flayed and horrible to see. And if this did not prevent you, perhaps you might not be able to draw so well.

Do you know what you will achieve if you practice drawing with a pen? It will enable you, trained and skilful, to draw a great deal in your head.
Exert yourself and take delight in copying always the best things – crafted by the hand of great masters – that you can find.

Cenninno Cenninni, Libro d’Arte, 1400

In the doctor’s schools of anatomy he dissected the corpses of criminals, undismayed by the brutal and repulsive nature of this study and only eager to learn how to portray in his painting the various limbs and muscles, their bending and stretching, in accordance with the laws of nature.

Paolo Giovio, 1525

I would not like to neglect to repeat the words I heard King François I say about him: (…) he did not believe there could be anyone else on earth who knew as much as Leonardo, not just about sculpture, painting and architecture, but also insofar as he was a great philosopher.

Benvenuto Cellini, 1562

Altogether, his genius was so wonderfully inspired by the grace of God, his powers of expression were so powerfully fed by a willing memory and intellect, and his writing conveyed his ideas so precisely, that his arguments and reasonings confounded the most formidable critics.

Giorgio Vasari, 1568

During his apprenticeship, Leonardo sometimes made clay models, draping the figures with rags dipped in plaster, and then drawing them painstakingly on fine Rheims cloth or prepared linen. These drawings were done in black and white with the point of the brush, and the results were marvellous, as one can see from the examples I have in my book of drawings.

Giorgio Vasari, 1568

But before we go any further, we must say a little more about Leonardo’s personality and talents. The many gifts that Nature bestowed upon him concentrated themselves primarily in his eye. Hence, although capable of all things, he appeared great above all as a painter. He did not rely simply upon the inner impulses of his innate, inestimable talent; he permitted no arbitrary, random stroke of the brush; everything had to be deliberate and considered. From the pure proportions to which he devoted so much research, to the strangest monsters that he compiled out of contradictory figures, everything had to be both natural and rational.

Goethe, 1787

It requires a quite different strenght and agility to maintain one’s position within a system that is never fixed and where ideas are free and still evolving, than in a dogmatic world. Leonardo da Vinci stands higher than Michaelangelo, Michaelangelo higher than Rafael.

Nietzsche, 1885

Painting, for Leonardo, is an operation which calls for every sphere of knowledge and almost every technique. Geometry, dynamics, geology, physiology. Representing a battle requires a study of whirlpools and swirls of dust; he will only portray them having observed them with his own eyes, so that his attempt will be well researched and informed by an understanding of their laws.

Paul Valéry, 1895

Art and scientific genius came together in Leonardo’s spirit.

Thomas Mann, 1936

Tolerance, it is true, demands that we respect differences of belief. But as already recognized by Leonardo da Vinci, to the degree that the truth becomes better known, so general consensus will come to replace individual opinions.

Thomas Mann, 1945

______________

[1] Source: Leonardo da Vinci, The Complete Drawings by Johannes Nathan and Frank Zöllner, Taschen, Bibliotheca Universalis, 2023

True Machine Reasoning

Proper, scientific machine Learning.
Proper, scientific machine Reasoning.
The engineering Middle Ages ends today.

Arithmetic and Geometry offer the supreme certainty.

Leonardo da Vinci

The presentation, without the demonstration, can be downloaded here.
You can watch the video on Rumble here.

This model is based on the proper, scientific machine model of Learning which is described here.

February 2026 Update:

This model became the basis for a proper, scientific machine model of Cognition which is described here.

____________
Header image: citological drawing by Ramon y Cajal.

The pyramid

I have built a pyramid from scratch.
Starting from the height of the blocks (0.5 cm) and the desired width of the base (7.5 cm) and height (5.5 cm) of the pyramid, I computed the width offset between layers and with it the 2D dimensions of the blocks in each layer.
I used 48 blocks with different dimensions stacked in 9 layers.

The calculations are as follows:
h’ divides h in as many parts as x’ divides x.
Therefore, h = k h’ and x = k x’ with k natural and equal to the number of blocks in height.
Therefore, x’ = x / k = x h’ / h.
In a pyramid, the base side will be L = 2x, so x’ = L h’ / (2 h).
The width of the n-th layer will be L(n) = L – 2 x’ n.

The dimensions of the block in each layer depend on the architecture which I chose to be such as to preserve the maximum number of blocks with their original standard dimension which is 2.5 x 2.5 cm. This architecture gives 3 types of blocks which are the standard one, the one with one dimension dependant on the layer and another one with both dimensions dependant on the layer.

I made the blocks with air dry clay and I used white cement and ocher yellow powder for the external cover.

True Machine Learning

While the Medieval Age of engineering feasted on the misinformation and disinformation about machine learning and artificial intelligence, I have been addressing the real problems.

Today true machine learning is born.

Today the Dark Age on this subject has passed.

True Artificial Intelligence is now being woven, a beautiful child that will come forth from the thoughtful and artistic womb of dutyful posterity.

For a context about this subject please refer to my 2019 article, since my position hasn’t changed since then. You also benefit from reading this other article.

The presentation, without the demonstration, can be downloaded here.

The communications to the Portuguese Instituto Nacional da Propriedade Industrial, the European Patent Office and the United States Patent and Trademark Office can be accessed by clicking on the respective links.

presentation
demonstration

You can watch the videos above on Rumble:
True Machine Learning
True Machine Learning – Demonstration

March 2025 Update:
This model became the basis for a proper, scientific machine model of Reasoning which is described here.

____________
Header image: citological drawing by Ramon y Cajal.

Leadership and a Violin

The process of developing new skills while we are already mature enough to assess that development itself may lend to an infinite loop of added value. You are challenged. You adjust yourself. You explore the challenge by relating it with previous experiences. You map the universe you know to the new reality.
The effort is crowned when you discover what it means to have learned something new towards other dimensions in which you are still developing.

What can this mean in the context of ______?
What have I learned about this that can help me navigate ______?

This article illustrates this point – that you can exponentialize your learning by assessing your learning process itself and by devising its potential applications.

Leading with the right hand

The right hand sets and keeps the strings in motion by passing the bow hairs over them. Because the hairs are powdered with rosin, this friction energizes the strings. Each string will vibrate at a specific frequency at any given situation, depending on the tension at rest, the vibrating length and type of string (each string is different in the way it is built or the material used).

In what concerns the right hand, this frequency is not determined by the violinist.

In the most simple case, in which we do not want to add ornaments to the notes (intensity, tonality, chromatism, etc, which we collectively term embelishments), the role of the right hand relative to the string being played is merely to set it in motion and keep it doing so for the duration of the note. Because we want the natural sound, we want to let the strings vibrate at their natural frequency.

The right hand will be imposing nothing more on the string than merely the mimimum friction to keep it vibrating.

Anything else we add to this, relative to the string, will change its natural sound; it will either be an ornament or a deviation from the intended sound.
Its also up to the right hand to choose the string being played.
These two roles mean that the right hand will be changing its position to set the bow angle to tackle the right string, and it will be varying speed depending on the time value of each note. Regarding the angle of the bow relative to the violin, there is a very limited degree of freedom between adjacent strings, and transitioning between non-adjacent strings requires a precise knowledge about this range a priori.
Changes in speed of the bow are especially relevant when also changing direction of stroke or crossing strings, in which case a similar situation occurs as when starting the first note (at the beggining of the music or after a pause).

Start with the miminum effort to set the strings in motion and only right after concern yourself with setting the right speed of the bow.

For this you start by moving not your arm, but just your fingers or in an even smallest degree your wrist.
So there are two key attributes of the right hand leadership at play here, which it is very easy to conflict with each other when learning:

  • precision
  • smoothness

You know precisely what your movements will be just before executing them, but that intentionality and precision are put in place with the minimum stress on the strings.
I use mental frames to set me up for doing things in specific ways, such as in this case. I picture my hand resting on the bow at all times, not commanding it. To get a notion of what it is, rest your forearm in some surface where your hand and wrist will be free to move. You hand will drop below the line of your forearm and your fingers will be slightly curved towards your thumb.Now place a pencil between the thumb and the other fingers, grab it with the tips of your fingers and they will almost exactly land in the usual relative positions violinists hold the bow.

The most confortable, natural and effortless disposition will produce the best approximation to the proverbial “pure and singing tone”.

This is a principle that guides formal composition and transcription as well, because there are countless ways a given sequence of notes can be played, and all decisions in this respect should place the musician in the most natural position for execution:

  • the notes can be played in separate dedicated bow strokes or in the same one (legato);
  • the same note can be played in different strings (with different fingerings);
  • the difficulty of added ornaments may depend on combinations of the previous two factors.

Leading with the left hand

The left hand sets the vibrating length of the strings, by pressing them with the fingertips against the fingerboard, which in turn determines the exact frequency of the sound produced when playing, along with the tension of the string at rest and the type of string.
The key attributes of the left hand leadership are

  • strenght
  • anticipation

The finger has to fix the extremity of the vibrating string acting like a small hammer for the duration of the note. Less pressure than that will slow the intended frequency of the note for allowing more freedom of movement to the string and will sound the same as a muffled lower pitch note.

This attribute of the left hand contrasts with the right hand, as you may have noticed already, and that is an added difficulty when learning because the two hands behave very differently but in precise concert with each other.
As for the anticipation, it is two-fold

  • The left hand executes before the right hand
  • The available left fingers anticipate the next notes

This means that the bow plays on a string which is already stopped by a left hand finger and that, while playing a note, the next notes should be anticipated whenever possible so that this principle is applied for two consecutive notes. That is, while playing a given note, if the next fingers you will be using are available, place them over the strings before you need them. This may lead you to play on a finger while having up to three other fingers already in place (most probable if the notes are very short in duration). This in turn will allow you to focus more on the bow between notes, which is specifically useful when crossing strings.




In this image you can see the same sequence of notes in time, the notes stopped already by the left hand fingers in waiting (orange) while the green note is being played.

Without the left hand leadership, a melody is played in a succession of disconnected individual notes, and these will start with a delay between them making the start pitch of the note always lower than its final one. On the opposite, provided that all other factors are adequately accounted for, the 2nd dimension is what glues everything together into a natural performance, much like water flowing in a stream, because

it sets the stage for reducing the overall effort of playing.

So there you have it:

  • precision
  • smoothness
  • strenght
  • anticipation

Alexander Markov’s beautiful performance puts it all together in this excerpt ending with a timely advice on the subject [1].

[1] Alexander Markov plays Paganini’s 24th Caprice, an excerpt from ref [2], and talks about violin practice from ref [3]
[2] https://www.youtube.com/watch?v3DaGGPEDWQA5A
[3] https://www.youtube.com/watch?v3DNSoWTWXoIwY

____________
This article was originaly published on LinkedIn at August 10, 2021.

Motivation

Suggested listening:

When I am in the sea I seldom focus on the people there. For obvious reasons. There everything happens at the rythm of the waves, of the sets of waves and of the tides. Its not about people there. People are the aliens in there. For that reason, and because of a couple more reasons that I will illustrate here, I rarely give advice to anyone there. Despite having lost count of the time I spent there.
One day the setting was inviting some sort of meditation. Small waves. Long wait between sets. Many people there learning the job. I spent some time waching them. After all this time it is straightforward to spot what’s going on in the people’s minds there, in general and most particularly if they are below a more or less arbitray degree of proficiency. You see the ones that are confident. Or uneasy. Or affraid. Or angry. Or whatever. The body posture, the face, the positioning in the lineup and on the board. They don’t lie.
There was a person there who stood much time still, hands grabbing the board, nervously looking at the horizon, each wave a decision proof. The decision fled away into perceived safety everytime.
At some point I focused on the coming set, after a thought I told her – this wave is going to be a good one.
She looked at me apparently making sure I was talking to her, then looked at the wave summoning in her direction.
She looks again at me, this time her eyes were lit up in confidence, she positions herself with the most hardworking focused posture – the complete opposite of a minute before – and starts to paddle. I thought to myself ‘not sure how this is going to end but let us see’.
She stood up and glided the wave.
I felt relief because it proved my assessment of the situation, despite knowing fully well how easy it is by now for me to anticipate the behavior of a coming wave. She then thanked me and I got on with my usual affairs.
There are these two things, at least, to take into account when you want to motivate someone.

  • did you assess the goal you are motivating people into?
  • did you assess the circumstances of the people you are addressing?

My bet here was in the match between the situation and the person. If your advice fails to bear fruit, won’t it erode confidence in others? Can you really motivate someone without that knowledge? Without it, motivationals are just… empty words.

____________
This article was originaly posted on LinkedIn at March 2, 2019.

Systems Thinking. Why?

Suppose we are to assemble a team for important tasks. We want the best to do the best. Therefore, we gather

  • a very good computer engineer
  • a very good project manager
  • a very good analyst

It is important that they are good at what they do, because they will be facing challenges in their field and we want to make sure they follow the best practices for the known problems and deal proficiently with the unknowns. What do you expect to happen later on? The project should be a success, shouldn’t it? After all, the expected challenges were dealt with and the unexpected were tackled in the best possible way.
If you wondered, while thinking about this brightful scenario, that something is missing in this picture, you may be already thinking in a systems perspective.
What’s missing here is

What are those very good people working for exactly?

The matter is not what they are doing. We know already that

  • the computer engineer should be making good engineering
  • the project manager should be doing good management
  • the analyst should be doing good analysis

Since what is being done intersects different areas, we can put the team doing precisely something that should not be done, while still being able to claim

I am following the best practices of my craft.

The systems view is the additional space created when relating different work fields in this example. Just like in mathematics, that new space is n-dimensional and cannot be fully described by partial combinations of each basic dimension (any n-k-dimensional space), and the same way, we cannot perceive the objects in the upper dimensional space by looking at each of the dimensions alone. This means that although we can reduce our analysis to each of those fields alone and still perceive some properties of the upper dimensional space, there will always be some color missing on the painting unless we take into account the system generated by those components. If systems thinking requires something from that computer engineer, or that project manager, or that analyst, is for them to have a look outside of their boxes. If we educate each them only about their craft, and later on confine them in their box, then we should not be that much surprised with those kinds of problems arising which we usually describe regarding the aplicability of systems analysis.
The same goes for

  • a politician
  • an economist
  • a military
  • a public relations

and irrespective of how advanced we may be in our understanding of

  • technology
  • science
  • religion
  • philosophy

and so on.
So, if you ever get unconfortable when someone finds some way of telling you

Just stay in your box.

you’re on the right track to figure out some new, unexpected and useful insights about otherwise ordinary affairs. And hopefully before you get unconfortable with the results of all that good effort.

____________
Image: from the movie Ex Machina by Alex Garland.
This article was originaly posted on LinkedIn at March 20, 2019.

Machine Learning or Machine Training?

There is a fundamental issue in statistical based “learning” systems:
i) they regard new facts as outliers in their known models
ii) they regard new facts not by themselves but by similarities with their known models
 
This is not a problem of course, when we want them to perform well and predictably in very specific scenarios which we know already by hand and which we believe will not change in time.
A face will still be a face, a road a road, a written letter a letter etc.
To this “learning” paradigm, “new different data” and “statistical outliers” are just the same thing. 
The problem here is precisely with Learning.
The statistical models do not learn with data — they are modeled or trained by data. And that only happens when we are building the model before actually use it.
So, the first time something new happens, they will deal with it the same way they have always been doing. Just like a cat cannot learn what is a human, so he can only interact with humans according to is notion of “others” which he will put in a pre-defined category in his mind such as “prey”, “provider”, etc.
The point is, we could train a cat to behave with humans differently than with everything else, but he was just being trained, not taught. He still could not learn anything new about people besides what he already knows.
 
About 10 years ago, I tried to solve a specific and apparently simple problem related to this.
I wanted to understand better how we humans can acquire so much information and seamlessly integrate it in our own mental processes, without being bogged down by the exponential complexity of the relationships that our own behaviors required us to learn.
I tried to understand how simple questions could be answered with simple responses, as soon as new data was acquired. I could not succeed in this. No matter how much I tried, the explosion of relationship combinations broke any heuristic every time. I quit.
Remember, statistical models can build complex relationships or mappings between datasets, but their answer is probabilistic and even the most simple questions will fail if entailing something not previously built in the model.

The explosion of relationships I mentioned happens because I am not restricting my data search on specific or preferential relationships. I wanted no heuristic guiding the search besides the data itself.
This assertion in an article about knowledge graphs in ForgeAI clearly states what I wanted to avoid, because it would condition every end result:
Because our edges are interpreted as probabilities, it is possible to set a probability cutoff beyond which we are not interested in graph connections. This allows us to only consider graph algorithms over highly restricted subsets of the graph, which provides us with major algorithmic improvements.
These probabilities, the cutoff, not knowing for sure if excluding a subset would exclude a more meaningful result, is precisely the problem I wanted to avoid. Basically, we would simplify the procedure by excluding “less relevant” information. This turns the process nondeterministic, which is a problem when you have to explain your results with anything other than a probability. In the worst case, I wanted to apply any heuristic over the general search method, to assure I was not biasing the results beforehand.
I am not sure wether you (reader) understand the scope of troubles we put ourselves into when we start doing cutoffs. Cutoffs are based on a judgement (eg some mathematical function someone chose), which in turn is based on relationships probabilities, which in turn are based on an heuristic, which in turn is based on a reasoning made ultimately by someone, no matter how data-informed it is. On top of that, if those probabilities are static, you are assured that at some point in time you will be doing the wrong assessments on data. If they are not fixed, then you have to build another machine to govern those probabilities in time.
All this happens and you are never quite sure that the best answer of all for any given query lies just two nodes ahead of one of your cutoffs. You have not exactly a major algorithmic improvement but a major algorithmic performance improvement, at the expense of the precision of the result.

Anyway a few years ago, I got back at it. In the mean time I had learned a bit more about mathematics, and I already knew a good deal of neurophysiology and anatomy. I also had nothing else really interesting to do.
But still nothing to do with what we call machine “learning”. I never liked statistics because it relies on a downgrade of each and every single individual case, which for me is the first thing I would do if I had decided to quit learning anything at all.
As for mathematics, it gave me a framework for thinking and exploring, for abstract work. Just that.
Like I said above, statistical “learning” has nothing to do with learning. Its about training. I hope I am being crystal clear about this.

This time around (a few years ago) I solved my 10 year old problem. I will not share the details here but I can say that knowing geometry, embryology and having time to spare did help (call it Learning).
Let me remind you that the problem was simple, and that was I was convinced it should have a simplistic solution, regardless whether it was hard or not to find (I believe it was).
Solving the exponentiality of relations meant also that having a huge database of facts stored should not impact the usage of it.
My point here is not about anything new, or innovative. It has already been done. My point is precisely that it can be done, without any prior knowledge of this field, and the end approach can be (at least technically) simple.
My point here is about “learning” and human versus machine capabilities. ItE28099s about the implications of the ways we approach problems and problem solving.

So, lets call this system George. George will show you that he can learn new things quickly.
Moreover, I am not telling George anything about the facts that he gets to know. This is relevant to those who are familiar with the usual heuristics, because they all require some “understanding” about the data and some previously defined way to use that “understanding”.
[As a note to those who are familiar with these subjects, I say that to weight a graph you need un understanding beforehand, which is not the case I am addressing here.]
Again, my point is that this is against the definition of “learning” in the first place.
George, I will say again, knows nothing, except one thing: he knows how to explore relationships without being overwhelmed in the process, whatever they may be (the problem I tried to solve in the first place).

I tell George “Alana knows Eve”.
I add that “Eve is Jay’s teacher”.
Now I can ask George how Alana and Jay are related (1).
He says:
Alana knows Eve.
Eve is Jay’s teacher

Simple, right? Of course. Its all Gorge knows anyway. 
Now George learns that “Jay was at Alana’s birthday”.
If I ask George again about how Jay and Alana are related, he should give me a different answer, because in the mean time he “learned” a more simple answer, right?
Now he says:
Jay was at Alana’s birthday

Great, George!
But lets put George to the test. He knows better. 
George learns that
“Leonardo is father to Alana”
“Eve bought is car from Alec”
“Leonardo plays soccer with Alec”

Now there is more than one answer to the question “How are Leonardo and Eve related?” (2).
George got it right again:
Eve bought is car from Alec
Leonardo plays soccer with Alec
and
Alana knows Eve
Leonardo is father to Alana

George also tells me that for him, the difference in these answers relies on their context.
He says that the context for the first is “Alec” and for the second is “Alana”. Go figure out why.

This is what George knows up to this point:

  • Alana knows Eve
  • Eve is Jay’s teacher
  • Jay was at Alana’s birthday
  • Leonardo is father to Alana
  • Eve bought is car from Alec
  • Leonardo plays soccer with Alec

 
Remember, the only thing I taught George to do, was to search the answers by himself, aiming to the most simple ones.
The principle in the first example (1) still applies — if I teach George a lot more about Leonardo and Eve, he will only consider it if it simplifies his answer for a particular query.
But now you could think — if more knowledge doesn’t “degrade” the answer’s quality, does it degrade its performance?
Lets raise the bar for poor George.
So George goes to college. There he learns a lot more. Ready to put it to practice, just like we humans would like to be able to do! 
So he learned a lot more about Leonardo and Eve. But also, many other not necessarily related things (but potentially related).
We ask him the same thing again and check is answers and how long he takes to get there.

Remember, each time the database changes George is unsure whether the old answers are still good enough.

Life is tough on George, and so are we!

Besides the previous information, I added 6000 random relations between another 99 people, including with the first persons we used. So the previous names are now related in many more ways and with many more people.
Againd the question (1), and the answer takes the same time (some ms):
Jay was at Alana’s birthday
 
But this time, he has other answers he learned (each line is an answer):

Some random assertion relating Alana, Natalee, Asa, Lisa and Jay
Some random assertion relating Jairo, Erica, Jay, Jeramiah and Alana
Some random assertion relating Jay, Alana, Molly, Anne and Eve
Some random assertion relating Jay, Alana, Moriah, Charlee and Lindsay
Some random assertion relating Jay, Kingston, Alana, Aldo and Noemi
Some random assertion relating Alana, Natalee, Asa, Lisa and Jay
Some random assertion relating Jay, Alana, Molly, Anne and Eve
etc

He is equally fast giving them. Take notice that all answers are direct relations between Jay and Alana. There are many more kinds of relations of course, but George wants to make it simple so he’s not adding more than we wanted to know about.
Again the question (2) about Leonardo and Eve (each line is an answer):

Some random assertion relating Leonardo, Logan, Marcus, Destinee and EveSome random assertion relating Eve, Leonardo, Jairo and Dashawn
Some random assertion relating Dashawn, Hailey, Eve, Angel and Leonardo
Some random assertion relating Leonardo, Erica and Eve
Some random assertion relating Eve, Landen, Kelsie and Leonardo
etc

Remember that the initial answers were more complex (two related assertions). George still simplifies things without additional delay.

Unlike a statistical model, he adapted his behaviour to what he learned (without training), and moreover, knowing more made things easier for him.

 Now lets find some more complicated relations.
I ask George about the following persons (3): Lillianna, Douglas, Allan, Rosemary, Libby and Kristina.
He answers (each 3.x is an aswer):
3.1)
Some random assertion relating Natalee, Douglas, Anne, Charlee and Lillianna
[On the other hand]
Some random assertion relating Lillianna, Jamari, Libby, Allan and Jairo
[Following]
Some random assertion relating Rosemary and Lillianna
Some random assertion relating Kailee, Kristina, Lillianna, Campbell and Logan
 
The “on the other hand” indication means that the assertion is not related directly to the previous one, but via the assertion “following” it. It’s just how I chose to express it.
3.2)
Some random assertion relating Salvador, Douglas, Logan, Leonardo and Lillianna
[On the other hand]
Some random assertion relating Lillianna, Jamari, Libby, Allan and Jairo
[Following]
Some random assertion relating Rosemary and Lillianna
Some random assertion relating Elizabeth, Kristina, Lillianna, Bridget and Salvador
etc
 
There are other answers, of course, but George is quick to explore them.
However, there are no simpler relationships than these — 4 assertion responses in a universe of over 6000. If less than 4 assertions would be a valid response, he would give them. Remove one of the assertions in any of those responses and you get a wrong answer.
I can tell you that George could know entire book libraries and still he would not stumble to give short answers. Naturally, in that case, to give all the answers would take more time, but I claim it is supra-linear in complexity , not exponential. 
You may argue, that this is just some simple search on data relations. If that is your opinion, then can you guess which algorithm is used? In these examples I am not using asynchronous techniques and the implementation is not fully optimized. I am using middle grade hardware.
 
Back to my point with all this.

If George was a statistical machine, he would make up an answer, the one that fits better to his model parameters.

No single learned fact could ever change radically his behaviour.
And his answer would not explain its own relevancy to the problem we are trying to solve.
Everything outside his scope of training would be reduced to a probabilistic similarity, therefore adulterated. He was not learning but instead he was changing data to fit what he already knows.
But we do not want that. We want to know how George got to his answer, or why any answer is relevant to our query, not how related it is to what he was trained to do.
This is the trademark of Learning — the system changes its state and its behaviour in response to changes in its environment.
In the first place, we didn’t give George any parameters about the data we fed him. If we gave George an heuristic, he could do the same thing he is doing but only focusing on the information that best fitted that heuristic (again the problem with statistical models). But anyway, filtering data is the easiest part and it would make things easier for George. Not my concern here, as you might have guessed already.

Intelligent behaviour explains itself.

If you were on trial for an offense, and the judge was to give you a sentence according to machine learning, he would say:
— Since the N similar cases in our database were found guilty, I declare you also guilty. Because the “AI” “learned” from millions of similar cases and decided that way.
So, it was not your case that was on trial, but the average case. Do you understand now better the difference between intelligent decision making and statistical decision making? You better do before you start relying in statistical decision making processes in your organization or in your life.


Update:
In case you who are reading this don’t go directly to my point here with “George”, so that you do not be “stuck” to the simplistic examples I gave (relation between assertions) and miss it, I made this update.
If you want a machine that learns, it has to be able to gather information and relate it to answer queries or do something with it. For example, you would an intelligent system to be able to relate the propositions “I bought my car from Fred” and “Fred sells cars at Bristol” to know (or to have learned that) “I bought my car in Bristol” without knowing it beforehand. And that system should tell me why it said that: because of those facts that it learned. This is a requirement to build knowledge and emulate rational or intellligent behaviour.
Look at it this way: a statistical machine (our today “machine learning”) relies on the ability to derive a state from a great number of tried possibilities (loosing a lot of information in the process). But a combinatorial kind of intelligence derives its power from the exponentiality of the number of combinations, that can be tried on the fly without previous training. The correlation between data that a statistical system does on its training, is done on the fly by a combinatorial system. Moreover, it is done not as a static framework but as a part of a dynamic evolving system. The knowledge in all those cases used in the current ML trials is implicit in the combinatorials that can be inferred from a much simpler state.
Of course, once a system is built that can not only internalize new information, but create new data from it, the process feeds itself and will surely get “out of hand”… at least human hands. Hope you do not feel as uneasy with it as I do!

The moral of this story is the following.

One of the things that makes us humans is this ability to really learn.

Not just to be taught or trained to do or know something — but to Learn something. That is, taking in something new and deriving value from it by relating it to what we previously known.
I share this example with you to at least motivate you to think about this:

  • are you really learning something?
  • are you really producing knowledge or just reproducing it?
  • are you really solving problems or just following a guideline?
  • are you assuming things or are you questioning them?
  • do you know exactly what you know?

In the end…

  • are you being human?

If you do not think for yourself, be sure that someone, or something else, will eventually do it for you.*

As for feelings, of course machines cannot feel, but it is a fact that we cannot build knowledge from feelings (although we love to feel we know something).
I wanted to share this case with you to help you understand this bottom line (*).
 Hope I succeed and contribute to a more critical and informed mindset about the most recent technical advancements and where we are heading in to.

____________
This article was originaly published on LinkedIn at December 31, 2018.