Mind the Math - The CSO Method

Behind the constant droning of race cars, wheel guns, and service equipment lies a seemingly unending stream of information. From the cockpit of the car to the pit wall, and from the paddock to the war room, countless personnel work tirelessly to ensure that each component of the race car is being used to its maximum capacity towards the ultimate goal of improving consistency and ultimately driving down lap times. WPE’s unique capabilities as a data-first engineering operation have been refined through high pressure, high impact successes in the motorsports and aerospace engineering sectors. From this, WPE has developed a framework that focuses on utilizing the full data lifecycle to bind together and bring to the forefront all of the impossibly faint patterns that determine a team’s success. Woven together like a fine tapestry, and wound tight like a watch, each data stream must be perceived, measured, and evaluated with respect to all others. The method derived by WPE is broadly conveyed with the acronym CSO - Cluster, Shift, and Optimize.

Cluster

It is not hard to consider a car as simply an assembly of many smaller assemblies, recursively expanded to the resolution of a single individual part like a bolt or a brake pad. What does tend to be difficult is to consider these assemblies not by their physical makeup, but by their influence on performance. Instead of thinking about the right front suspension or the underbody aero package as an assembly of components, think instead of the groups of components which affect turn-in brake stability or power-on steering response, even when these components are not physically coupled to one another. We refer to the former as a component-based model and the latter as a performance-adjustment model (PAM). We do this for the simple fact that it is not the group of parts that we typically care to improve, it is instead the quantitative and qualitative response characteristics of the vehicle. The beauty of modern motorsport is that we often have the data needed to make this change, as long as we know what to do with it. The clustering process is a multi-stage unsupervised machine learning pipeline which takes in telemetry and vehicle dynamics model data and in return provides a re-mapped model of the vehicle where components are no longer grouped (clustered) by physical location or type, but by the aspects of vehicle performance which they directly influence. This new concept of vehicle modeling can be scaled such that nearly any conceivable performance concern can be constructed as a cluster of relevant vehicle components, and even non-vehicle components such as pit stop operations or weather conditions.

Beyond the re-clustering of components into the new model, there is a further layer of categorization that classifies each component in terms of the nature and extent of its adjustability. This is a critical step to prepare for the shift operation. Below is a simple diagram that represents the transformation. Note that it is intentionally and substantially simplified.

Shift

When the clustering process is complete, the vehicle is fully modeled as a series of interrelated performance metrics, represented as nodes, and adjustable components, represented as edges which connect these nodes. The next step is to determine where the sensitivity lies in this ‘vehicle model network’, which is to say what edges have the greatest influence on each node. Once this is known, an iterative process begins wherein the sensitivity is shifted away from components that cannot be adjusted readily, or cannot be adjusted within a reasonable degree of certainty at the track. The intent is to provide as much sensitivity as possible within the constraints of readily accessible on-car adjustability, such that the crew at the track have the ability to make meaningful setup changes should the conditions necessitate it. Take for example a NASCAR Next Gen race car, which has a myriad of mounting options for its control arms as well as deeply complex damper adjustability. If a setup were unloaded at the track which utilized a substantially extreme control arm configuration, such that the dampers needed to be near the edge of their range of adjustment to maximize traction, only a small disruption to the track conditions could risk placing the crew in a scenario where they no longer have a performant setup, nor the appropriate sensitivity to compensate with damper adjustments, nor the time to remove and re-mount the control arms. Had there instead been a control arm configuration which placed the dampers near the center of their adjustment window, the crew would find a much wider range of adjustment available to them given their time constraints. In a more extreme example, consider a WEC program that develops an extremely stiff LMH chassis such that nearly all compliance must come from the springs and dampers. This may be suitable under some conditions, but it is the nature of many tracks on the WEC schedule that there is often rain which very clearly disadvantages excessively stiff cars. A ‘softer’ car may be able to find a very strong all-around setup that will work well under mixed or changing conditions, while the stiff-chassis car would be forced to make substantial sacrifices in some aspects of performance to make up for the lack of adjustability available to the crew. This has been seen in recent years with some of the lower-class prototypes falling behind the GT cars in pace, with seemingly no ability to compensate during the course of the weather event. While this is typically the most human-centered aspect of the CSO method, due in large part to the reliance on the expertise of race engineers and drivers, it too can be an exercise in mathematics and statistics if there is sufficient performance data and/or a fully-featured vehicle dynamics model. Very large scale neural networks can be used to model the changing relationships between the edges and nodes of this ‘vehicle model network’, and to determine optimal configurations for major non-readily-adjustable systems such that sensitivity is shifted towards more readily adjustable components. Once this is complete, the optimization process can begin.

Optimize

With maximum adjustability placed in the hands of the crew, the final stage of the CSO method is to determine the specific initial value that each adjustable component will take. This process is the most computationally-centered aspect of the CSO method, and relies heavily on the use of well-tuned optimization algorithms. Broadly, optimization falls into two categories depending on the component type; ‘optimal design’ and ‘robust design’.

Optimizing for an ‘optimal design’ is the mathematical procedure of determining what exact value an adjustable parameter must take to maximize some value function (like average speed) or minimize some cost function (like lap time). This is the most commonly understood definition of ‘optimization’, and is that which most people are engaging in when making setup determinations.

Optimizing for a ‘robust design’ is a more nuanced variant of the above procedure where value is given to maintaining the ability to make small but meaningful adjustments, even when it sacrifices outright performance as measured by the value or cost function. This is a deeply misunderstood method of optimization in the motorsports industry, but is a key tool for those teams who seek to find consistent success.

Those not familiar with optimization algorithms may wonder why there is a concern for two different optimization paradigms. For an example, consider a budding sportscar program that arrives to the track with every adjustable component centered within its range of adjustability. This is a thoroughly robust setup, and affords the team a great degree of adjustment for any possible driver feedback, but it is highly unlikely to be the best car on track. This is not uncommon for new sportscar teams, who resort to a single ‘baseline’ setup from which track-specific setups and data books can be built over time. Now consider instead a top-level program that uses state-of-the art lap simulation tools to find the singular combination of adjustments that results in the mathematically ideal lap time. This is a thoroughly optimal setup, but with no regard for adjustability they may find that their car performs poorly if any one aspect of their simulation was not representative of reality, and that they can do little about it.

Thus, there must be some balance struck between optimizing for truly optimal performance and otherwise for robustness of operation. In an ideal world, our ability to model and simulate vehicle performance would be so advanced that robust design is not a concern. In reality, there are many factors that may cause our simulations and models to be inaccurate, hence the need for a robust setup. The most common of these are simulator-vehicle correlation issues and unexpected weather patterns. The image below shows a generalized plot of the relationship between some arbitrary ideal lap time and some arbitrary starting (cold) tire pressure. If the model upon which the optimization algorithm ran was perfect, we would always chose the optimal point tire pressure, but if there is any chance that this relationship is subject to change or that it is not modeled with certainty, we may chose the robust point tire pressure to allow us to be adaptable in-situ. Note that this process is independent of that discussed in the Shift method detailed above. The purpose of the Shift is to provide a configuration with a wide adjustability range in the most general case, while the purpose of the Optimization is to determine what subset of that range is most useful for a given set of conditions or parameters.

To perform the above optimization methods typically involves a combination of non-gradient-based numerical optimization methods as well as some simple gradient-based approaches. Optimal design tends towards the former due to the ability of non-gradient-based approaches to find global optimums in noisy, sparse, or disjointed data while robust design tends towards the latter, owing to the necessity of determining your ‘local sensitivity’ in a robust design. In either case, care must be taken to introduce penalty functions that functionally exclude setups which may be infeasible for reasons other than on track performance, such as illegally low ride heights or ride heights that make a car unable to be jacked up if using a manual jack. This results in a ‘fault tolerant’ or ‘fault avoidant’ optimization paradigm.

Other Notes

There are three other points that must be made when introducing the CSO model; the general determination of how to adapt to CSO, the dependence on computer modeling and vehicle simulation, and the differences between machine learning and optimization.

To adapt to CSO it is important to understand where the currently-used methodology fails. If you regularly know that you need to make an adjustment, but you don’t know what needs to be adjusted to make an improvement, you have a failure to cluster. If you regularly know what needs to be adjusted, but you aren’t able to make that adjustment due to some external (time) constraint or the limitations of the mechanism you are trying to adjust, you have a failure to shift. If you regularly know that you need to make a small adjustment, but every small adjustment seems to worsen performance and not improve it, you have either a failure to balance optimality versus robustness, or you have simply maximized the total performance capacity of your platform (congrats!). We all wish it were the latter, but it almost never is.

While the CSO methodology was derived for the computationally-advanced environment of NASCAR/Prototype racing, it can of course be used in other contexts. The major concern becomes the ability to collect enough data to sufficiently train/develop the numerical models upon which the CSO method relies. It is an unfortunate truth that with decreasing data set size comes increasing mathematical uncertainty. A core focus of WPE is the development of low-order modeling techniques that allow for advanced numerical methods to be applied to smaller and less robust data sets, such as those seen in junior formula, late model, and entry-level sportscar programs. This lends well to the following section.

The terms ‘optimization’ and ‘machine learning’ are often conflated within the motorsports industry, but they are in fact two separate ideologies. Machine learning is the process of constructing an analytical model that can relate specific known inputs to specific known outputs. This process of ‘fitting’ results in an analytical ‘machine’ that can then be used to predict outputs given any set of inputs. It is a predictive modeling method. The ‘art’ of machine learning is in improving a model’s accuracy, avoiding underfitting (low accuracy, even for known input/output) and overfitting (high accuracy for known input/output, but low accuracy when predicting the output of a novel input). Optimization is the process of determining which set of inputs will result in the mathematically superior output, given the ability to look-up or directly calculate the exact output that would be produced by any given set of inputs. It is a prescriptive modeling method. The ‘art’ of optimization is in rapidly iterating through possible inputs to find that which is superior, often when there may be many millions (or more) of possible input sets. In short, machine learning will tell you that ‘if you do a, b, and c, expect an outcome similar to z’ while optimization will tell you ‘to achieve the best possible output z, you must do a, b, and c’. These two ideologies should not be confused, however they are not mutually exclusive. There are many cases where it may be advantageous to execute machine learning and optimization tasks in parallel. One such case is that where a race team has lap times at some specific track for some amount of vehicle setup configurations. There may not be enough data to optimize for a specific setup, but it may be enough to build a rudimentary machine learning model which can make rough estimates of how certain adjustments would affect lap time. The team uses the machine learning model’s output to determine the next setup to install on the car, and the corresponding lap time data is fed back into the model, which is subsequently re-fitted. At a certain point, the machine learning model will serve to point the team in the right direction, while the optimization model will serve to tell them how far to go in said direction. All the while, each real lap time collected is used to improve both models in a symbiotic manner. This system would, in principle, converge to a pure optimization model as the ratio of ‘real data points collected’ to ‘all possible data points’ increases asymptoticly to one. It is of course infeasible to ever reach a pure and complete optimization model, but nonetheless this analogy demonstrates the manner by which a skilled engineering operation can make use of both technologies efficiently.