The user interfaces we see when we browse the web are becoming increasingly intuitive and performant. Now, we have to think about how to support users with faster screen responses and smoother designs. But are we thinking in the same way about how to protect our users from scenarios we did not plan?

This article is not about error screens or user experience. It is about how development-level decisions can protect our customers from impossible scenarios and give the developer more confidence while writing code. Where can we start this journey? At the foundation of modern web applications: requests to the backend.

Trusting at Second Glance

The Problem of Trusting Too Early

Let’s imagine a scenario where, after (sometimes long) discussions, the backend and frontend teams have aligned on how requests are made and what data will be returned. Now it is time to implement the agreed contract. As an example, let’s think about a request to fetch a product data containing the price and the tax to be paid, calculating in the frontend side the total amount using those fields.

// Request of the product to the Backend
const myRequestedProduct = await fetchData('mywebsite.com/product/123');
 
// Get values from the requested product
const { price, tax } = myRequestedProduct;
 
// Show to the user the price, tax and the calculated total amount
<section>
  <p>Price: {price} €</p>
  <p>Tax: {tax} €</p>
  <p>Total amount: {price + tax} €</p>
</section>

What happens in the code above if there is a misunderstanding in communication and the backend, instead of sending price, ends up sending originalPrice? In this case, we would stop showing the product’s original value, and we would also compromise the total amount calculation.

Price: €
Tax: 12 €
Total amount: NaN €

One approach frequently used by developers to map what can be sent by the backend (and also ensuring the types of each variable in our application) is to use TypeScript on the frontend, which gives us more confidence that we are using the correct properties in each variable. However, it cannot be treated as a guarantee that what is being sent by the backend matches the defined type, since it does not perform those validations at runtime in the browser.

// The type definition of what Backend will send
type Product = {
  name: string
	price: number
	tax: number
}

// Request of the product to the Backend, 
// trusting that we will receive Product type
const myRequestedProduct = await fetchData<Product>('mywebsite.com/product/123');
 
// Get values from the requested product
const { price, tax } = myRequestedProduct;
 
// Show to the user the price, tax and the calculated total amount
<section>
  <p>Price: {price} €</p>
  <p>Tax: {tax} €</p>
  <p>Total amount: {price + tax} €</p>
</section>

Price: €
Tax: 12 €
Total amount: NaN €

// The type definition of what Backend will send
// now fixing the originalPrice attribute, but expecting a number
type Product = {
  name: string
	originalPrice: number
	tax: number
}

// Request of the product to the Backend, 
// trusting that we will receive Product type
const myRequestedProduct = await fetchData<Product>('mywebsite.com/product/123');
 
// Get values from the requested product
const { originalPrice, tax } = myRequestedProduct;
 
// Show to the user the price, tax and the calculated total amount
<section>
  <p>Price: {originalPrice} €</p>
  <p>Tax: {tax} €</p>
  <p>Total amount: {originalPrice + tax} €</p>
</section>

Price: 20 €
Tax: 12 €
Total amount: 2012 €

Validating Properly What Comes

So how can we validate that the data is arriving in the right format and in real time? We can implement these validations manually, but there are also several libraries that offer different ways to validate if a variable has the correct types (e.g., Zod, Joi, Yup, etc.). For demonstration purposes, we will use Zod, but the choice is entirely up to the development team based on their own preference.

In this context of requesting a product from the backend, we should define a schema (similar to TypeScript types) with what is expected for each attribute, validate if the fetched content matches what is expected, and then proceed to show it to the user:

// Definition of schema with expected attributes and types
const ProductSchema = zod.object({
  name: zod.string(),
  originalPrice: zod.number(),
  tax: zod.number(),
});

// Request of the product to the Backend, 
// expecting an "unknown" data first
const myRequestedProduct = await fetchData<unknown>('mywebsite.com/product/123');
 
// Validate if requested product matches our schema
const validatedProduct = ProductSchema.parse(myRequestedProduct);
 
// Get values from the requested product
const { originalPrice, tax } = validatedProduct;
 
// Show to the user the price, tax and the calculated final price
<section>
  <p>Price: {originalPrice} €</p>
  <p>Tax: {tax} €</p>
  <p>Total amount: {originalPrice + tax} €</p>
</section>

In case of the expected type for the original price still being string and not number, when we validate the requested data using our new schema, an error will be thrown in the application informing that the data doesn’t match what was defined:

[
  {
    "expected": "number",
    "code": "invalid_type",
    "path": [
      "originalPrice"
    ],
    "message": "Invalid input: expected number, received string"
  }
]

From there, we can handle the generated error and make the best decision on what to do, based on the product’s scope and the user journey defined by the team:

• Show an error message to the user if it is a critical path
• Report it in an error tracking platform, counting the number of occurrences of this same issue
• Update the defined type to the new possible value if there is no room to negotiate contract changes
• Notify the backend team that the request is returning values that were not expected

Type Assertions Without Validation? Better Avoid!

Following the idea of using TypeScript to define the types of our variables and functions, it’s common to see types being assigned using type assertions without any kind of validation, often because the developer believes that, in that particular path, the variable really does have the correct type and attributes.

Still using the product example, let’s assume we have two possible product types: products that have a size and products that have a color.

type SizedProduct = {
  name: string
  size: 'S' | 'M' | 'L'
}

type ColoredProduct = {
  name: string
  color: string
}

// A generic product can be one of them
type Product = SizedProduct | ColoredProduct

From there, let’s imagine that we need to compose the product description using its name and size, formatting it in a function that retrieves the product’s information.

// Format the description of a sized product using the name and size
const formatDescription = (sizedProduct: SizedProduct): string => {
  return `${sizedProduct.name} ${sizedProduct.size}`
}

// Function that receives a product an tries to compose the product information
const getProductInfo = (product: Product) => {
  const description = formatDescription(product);
  
  // The rest of the function code should continue here ...
}

In the code above, we would already see a TypeScript error since the formatDescription function expects a product with a size, while the getProductInfo function is calling it with a generic product.

Assuming that, for some reason, it isn’t possible to specify the product type used in the latter function (normally we would change the type defined as the parameter of the function, but we’ll keep it as-is for example purposes), if the developer assumes that this flow will always have a sized product, they might use a type assertion when calling the function.

const getProductInfo = (product: Product) => {
	// Usage of type assertion to call the function
	// saying to Typescript to trust your knowledge that this is a sized product
  const description = formatDescription(product as SizedProduct);
}

MyProductName undefined

Trying to prevent this scenario to happen, instead of using type assertions, it is always safer to ensure that the parameter we are passing actually has the expected type. One way to do this is by validating if the product itself has a size before formatting its description.

// Format the description of a sized product using the name and size
const formatDescription = (sizedProduct: SizedProduct): string => {
  return `${sizedProduct.name} ${sizedProduct.size}`
}

// Function that validates if the product is a sized product
const isSizedProduct = (product: Product): product is SizedProduct => {
  return 'size' in product;
}

// Function that receives a product an tries to compose the product information
const getProductInfo = (product: Product) => {
	
	// Guaranteeing a flow for not sized product
	if (!isSizedProduct(product)) {
		// ...
		return;
	}
  
  // Description that has a validated sized product
  const description = formatDescription(product);
  
  // The rest of the function code should continue here ...
}

Explicit Paths Using Union Types

One way to try ensuring that users don’t see any unexpected scenario is to make explicit in the code what should happen for each possible value being validated. Let’s take a look at this case where, for products with a defined size, we want to show a specific information for smaller sizes and a generic message for the larger ones.

// Still working with sized product type
type SizedProduct = {
  name: string
  size: 'S' | 'M' | 'L'
}

// Function to get information based on product size
const getSizeInfo = (product: SizedProduct) => {
  if (product.size === 'S') {
    return getInfoForSmallProducts();
  }

  return getDefaultProductInfo();
}

In this example, we have a function being called specifically for size S products, while another function is called for the remaining sizes (currently M and L). Now, let’s suppose we introduce a new XS size and we want it to put it in the same path as size S, it would be natural to add it to the first condition, as shown below.

// Now updated with XS size
type SizedProduct = {
  name: string
  size: 'XS' | 'S' | 'M' | 'L'
}

const getSizeInfo = (product: SizedProduct) => {
  // Added XS check here
  if (product.size === 'XS' || product.size === 'S') {
    return getInfoForSmallProducts();
  }

  return getDefaultProductInfo();
}

This flow tends to work perfectly. However, we can use TypeScript as an ally to guide us on what to do when a new value is added to a set of possible values for a given attribute.

In the case above, after adding XS to the list of possible sizes, we would not see any indication that something needs to be updated in the function. The developer should have the responsibility to manually find every occurrence of this type and add the logic to handle XS sizes. Without any changes, the flow would remain the same, and the new size which should have a specific information defined would receive a generic one.

How can TypeScript help us here? By using switch statements with a default case that show to us when something was not declared.

// Same function but with new behaviour
const getSizeInfo = (product: SizedProduct) => {
  switch (product.size) {
    case 'S':
      return getInfoForSmallProducts();
    case 'M':
    case 'L':
      return getDefaultProductInfo();
    default:
      return product.size satisfies never;
  }
}

Following this approach, when we declare in the default case that the product size satisfies the never type, we are defining that there are no other possible values that haven’t already been set in the switch statement.

In the moment we add XS to the list of possible values, Typescript will show an error to the developer in the default case, indicating that the XS value was not handled properly. This forces the developer to explicitly define what should happen for the added value.

Type '"XS"' does not satisfy the expected type 'never'.(1360)

Once the new size is included in the switch statement, the error disappears and the behaviour becomes properly defined.

// Now with XS value
const getSizeInfo = (product: SizedProduct) => {
  switch (product.size) {
    case 'XS':
    case 'S':
      return getInfoForSmallProducts();
    case 'M':
    case 'L':
      return getDefaultProductInfo();
    default:
      return product.size satisfies never;
  }
}

This approach can be improved if the developer considers this flow as a critical one and wants to throw an application error when an unexpected product size is found. In that case, the default case will throw an error along with performing the type check.

// Function that receives the unexpected value and throws error
const shouldNotBeReached = (value: never) => {
  throw new Error(`Unexpected ${value}`);
}

const getSizeInfo = (product: SizedProduct) => {
  switch (product.size) {
	  case 'XS':
    case 'S':
      return getInfoForSmallProducts();
    case 'M':
    case 'L':
      return getDefaultProductInfo();
    default:
	    // Using the new function here
      return shouldNotBeReached(product.size);
  }
}

Protecting the Code to Protect the User

All the approaches brought in this article were not presented to be a book of mandatory rules that should be applied. The goal is to show some ways to protect our frontend code, so we can protect our customers from scenarios that should not happen in our application. More than any guideline included here, the most important is to keep the mentality of making clear in our code what is the expected behaviour on each situation to avoid unwanted surprises.

Users are the main reason for an application to exist. Providing a smooth experience, without any impossible scenarios or malformed information, is the best way to retain them.

For software development teams, the moment we stop learning is the moment our technical stack starts becoming obsolete. It’s not about how many languages or tools we know. It’s about how quickly we adapt when they evolve, get replaced, or transform into entirely new technologies.

Take the transition from Lightbend’s Akka to Apache Pekko as a practical example. Akka has long been a cornerstone for building concurrent and distributed systems in Scala and Java, underpinning many critical services. However, as the ecosystem evolved and licensing considerations shifted, our engineering teams migrated some of these services to Pekko, a modern fork of Akka that maintains API compatibility, while offering improved governance and community support.

The move to Pekko wasn’t just a technical migration. It required engineers to understand both the underlying actor model principles and the practical migration strategies, demonstrating how continuous learning enables teams to adapt to technological evolution without disrupting core business functionality.

Overall, for engineering teams working at scale, keeping up with this evolution is not optional, it is a necessity.

That’s why the most effective engineering organizations treat learning and development as a core pillar of excellence, not an afterthought. Training isn’t generic or disconnected, it’s intentionally aligned with the technologies, architectures, and real-world challenges teams face every day. Our Learning & Development team works closely with engineering leaders to design training programs that are connected to our technology stack, architectural principles, and business challenges.

Continuous learning isn’t a “nice to have.” It’s what enables teams to stay effective, resilient and ready for whatever comes next.

Generic Training is Not Enough

Traditional corporate training can easily become too generic for software companies.

We navigate complex business landscapes, where global demand meets unique technology challenges, so when it comes to training a one size fits all solution does not make the cut. Tailored training programs, unlike generic content, accommodates role-specific requirements and real-world application scenarios that make learning contextual and integral to performance.

In addition, neuroscience shows that active learning, where people have opportunities to discuss content, solve novel problems, apply concepts, and engage in peer-to-peer interaction, strengthens learning pathways. This reinforces the idea that transitioning from traditional training programs to tailored learning experiences crafts more transformative moments in people’s development journey throughout their careers.

Beyond one-off training events, which can be sporadic, flexible learning ecosystems help make development continuous and embedded in day-to-day workflows. At Blip, we’re proud to support that flexibility by providing the tools our people need to grow, aligned with their ambitions, learning styles, and personal development plans. It reflects our belief that every development journey is unique.

Investing in personalized learning experiences that reflect the complexity of our business and product roadmap contributes far more effectively to long-term success than off-the-shelf training ever could.

Building Training Around Real Engineering Problems

One of the core principles we apply when designing learning programs is that training must reflect the real complexity of production systems.

With that in mind, we’ve developed a catalog of technical courses built directly around the technologies that power our day-to-day engineering work. Rather than relying on generic content, we mapped our actual stack and identified the areas where knowledge gaps could most impact engineering quality and capability.

The result is a curated program covering everything from distributed systems fundamentals, such as Kafka and Kubernetes, to software craftsmanship topics like Functional Programming and Clean Architecture. Courses are delivered by internal engineers who bring first-hand production experience, ensuring that what is taught is grounded in real challenges rather than theoretical abstractions. Where internal expertise is complemented by external providers, we ensure the content remains aligned with our architectural context and engineering standards.

But technical depth alone is not enough. Engineers building products in our industry need to understand the domain they operate in — the rules, the language, and the business logic that shape every system they design. This is a foundational element of our learning programs, aimed at building the business literacy required to make better technical decisions.

From understanding core business concepts and terminology to exploring the principles behind our products, these courses bridge the gap between engineering and product/strategic thinking. When engineers understand why a system exists and how the business works, they are far better equipped to design solutions that are not just technically sound, but genuinely impactful.

Personalized Learning Paths for Engineers

Not every engineer needs to learn the same thing at the same time.

A backend engineer focusing on distributed systems will have very different learning goals from a frontend engineer working on performance optimization.

For this reason, our training programs are structured around personalized learning paths, adapted to our industry.

These paths are shaped by a few principles:

1. Goal setting: the key to design a fit-for-purpose learning experience is a needs-first approach. In our industry, where demand and innovation evolve rapidly, it can be tempting to jump into training before fully understanding the gaps, a common pitfall to avoid.
   
   To address this, we rely on internal “tech radars”: people across different areas of the business who act as lookouts for engineering teams’ learning needs. We work closely with them to understand which new technologies, platforms, and frameworks are being adopted, and where capability needs to grow. This iterative collaboration ensures that our learning initiatives remain relevant and keep pace with constant change.
   
   
2. Speed: as the value lifespan of technical skills progressively shortens, we recognize companies must accelerate their learning velocity to quickly keep up. Our learning experiences are designed with speed to skill in mind, actively investing in flexible formats people engage in, accessible and rooted in skills development through practice and experimentation rather than knowledge transfer alone. As learning agility becomes increasingly rewarded, we enable it with training options that allow for speed to learn in an effective way.
   
   
3. Monitoring: The adaptive nature of a solution allows it to evolve while remaining valuable over time, and learning is not an exception. Continuously assessing feedback, identifying improvement opportunities, and moving back to the drawing board if needed is crucial to keep refining learning that matters. Learning data is a powerful tool, not just afterthought.
   
   

Engineers Teaching Engineers

One of the most effective learning models in engineering organizations is peer-to-peer teaching.

A good example of this in practice is the internal training program we built around Fundamentals of Functional Programming with Scala. The motivation was straightforward: it was a skill that our engineers needed but one that was not yet deeply embedded across our teams. What made this particular case interesting, though, was the nature of the gap itself.

Most engineers are reasonably comfortable picking up a new technology or programming language. The syntax changes, the tooling is different, but the underlying way of thinking stays largely the same. Functional programming is a different challenge entirely. It asks engineers to shift their programming paradigm, to reason about problems in a fundamentally different way. That shift is significantly harder to make, and a generic off-the-shelf course is rarely enough to achieve it.

For that reason, we built the training around our own context. Rather than relying on abstract academic exercises, the course was grounded in real problems we face in our systems day to day. The examples felt familiar because they were familiar, as they reflected the kind of decisions our engineers in production code. That connection between learning material and real experience is what makes the difference between knowledge that fades quickly and long-term knowledge that shapes how engineers reason about programming.

Learning as Part of Engineering Culture

Blip’s commitment to fostering a learning culture is one of its core pillars. Each person is empowered to take ownership of their development, choosing the paths that best suit their needs, whether through external courses, books, learning resources or conferences. This flexibility supports the continuous growth of technical, communication, and professional skills. In addition, everyone has access to an internal learning catalog covering a wide range of topics, as well as other platforms with extensive content.

Having resources available is important, but it does not end there. What truly shapes a strong learning culture is the people around you. Working with passionate, motivated, and highly skilled colleagues who are always eager to engage in technical discussions or offer their own perspectives creates a multiplying effect that, combined with a growth mindset, has a real impact on both personal and professional development.

A practical example is Luis Matos’s experience over the past year. Through a combination of internal catalog courses and external learning resources, he had the opportunity to explore new areas of knowledge. At first, the immediate impact of these courses wasn’t entirely clear, beyond curiosity and a desire to strengthen his technical skills and grow as an engineer.

Looking back, however, their value became evident. These learning experiences went far beyond initial expectations, shaping how Luis approached multiple challenges throughout the year.

To be more specific, that was a recent analysis of how to store information consumed from a Kafka topic and then used by multiple services. Knowing Kafka fundamentals proved critical for Luis and the team when evaluating Hazelcast as a distributed cache solution for storing consumed data. In particular, understanding consumer-group rebalances, committed offsets and the side effects of forcing consumers to read the whole topic on each partition assignment helped spot hidden operational costs, where some partitions could be re-read unnecessarily after a rebalance, potentially increasing lag and bootstrap pressure. At the same time, by applying industry-standard architectural principles, they kept data access behind a stable API backed by an abstraction layer for the repository used to store the data. This allowed them to potentially switch the storage implementation as needed without impacting API callers, thereby reducing coupling between business logic and infrastructure, providing a safer path to evolve the solution in the future.

Had Luis not had the chance to use the time and resources to learn these topics, delivering these tasks would have been much harder. That is why an environment that consistently encourages growth, with vast learning resources and rich technical discussions among colleagues, is more important than ever.

As such, a clear understanding of the business, architectural trade-offs, and awareness of system-wide impacts across different design options when analyzing solutions are truly important skills.

The connection between learning and technology is a two-way street. Learning drives technological advancement while technology expands learning at a scale. This symbiotic relationship puts forward an intentional and continuous cycle of growth.

"Done": A Dangerous Word?

In Software Engineering, the word "done" might seem simple, almost obvious. In reality, it can be deceptively slippery. What’s "done" for a developer implementing a feature may not be the same as what’s "done" for QA, Product Manager, or from a delivery perspective. 

Misalignment around this single word can lead to wasted effort, unexpected bugs, a false sense of progress, or even customer frustration. As Ana Filipa Silva puts it, one of the Engineering Manager’s key responsibilities is to help teams define, communicate, and agree on what "done" truly means, so everyone moves forward with a shared understanding.

Why Definition Alignment Matters

When a team isn’t aligned on the definition of "done", subtle but impactful issues start to appear. A feature may reach production without full test coverage, documentation may be incomplete, edge cases might go unnoticed, and technical debt can quietly accumulate. From a delivery perspective, this misalignment can lead to planning the next work under the assumption that a story is finished, while the team is still fixing gaps. 

That’s why having a shared understanding is so important. A definition of this word might look like bureaucracy at first glance, but it’s not. It’s about predictability, trust, and quality. 

When every team member understands what is required for work to be considered complete, teams reduce rework, build confidence in their deliveries, and free up mental space for creative problem-solving instead of constant firefighting. 

The Many Faces of "Done"

"Done" doesn’t mean the same thing to everyone, and that’s not a problem by itself. Problems arise when these different perspectives are never made explicit.

The Role of the Engineering Manager 

For Ana Filipa Silva, as an EM, defining "done" is about clarity and alignment across the team and its stakeholders. It’s not something Engineering Managers set for the team. Instead, the team should guide this definition, and EMs should help to make it explicit, visible, realistic, and sustainable. 

A big part of the manager’s role is creating the conditions for good conversations to happen. That starts with psychological safety, allowing everyone to share concerns and thoughts, ask questions, and be transparent. Individually, people often already have an implicit sense of what "done" means. But when that understanding isn’t shared, misalignment emerges. To our EMs, it’s important to bring these assumptions to the table, encouraging discussion, and helping the team agree on a definition that reflects both quality expectations and delivery realities. 

Predictability is another key concern. Planning only works when "done" is understood in the same way by everyone involved. Rather than pushing for speed, Ana prefers commitments based on a realistic view of the work, including testing, documentation, validation, and anything else required to feel confident in the outcome. When "done" is clear, planning becomes easier, less stressful, and more trustworthy. 

More than just planning, this definition can also help navigate trade-offs. There are moments when teams need to make conscious decisions about scope, risk, or timing. The EM’s role isn’t to override technical judgment, but to challenge, help frame those decisions, make constraints visible, and ensure that any compromises are intentional and understood by the whole team.  

Finally, what "done" really means should never be set in stone. Teams evolve, systems grow, and priorities change. What was "good enough" six months ago may no longer be sufficient. Revisiting the definition doesn’t need to be a formal scheduled moment, but it should happen whenever the existing definition starts losing relevance. 

When this works well, "done" becomes a shared reference point that supports autonomy, quality, and confidence across the team.

The Role of Developers and QA Teams 

"Done" isn’t just about making the code compile or work on one developer’s machine, it’s far more complex than that. While getting a feature to function is an important first step, it’s only the beginning. 

A solution might seem to be working and still contain breaking edge cases, performance regressions, missing functionality, or small issues that will only be noticed later in the existing codebase. It may also introduce code that is hard to understand, extend, or maintain, slowly becoming a liability. If "done" is defined only by validating the happy path and confirming that "it works", critical aspects of quality are overlooked. Sooner or later, the system pays the price for those mistakes. 

From a developer’s perspective, as explained by Sérgio Gomes, Senior Software Engineer, "done" starts with meeting the acceptance criteria, but it certainly does not end there. Delivering a feature also implies that we meet our engineering standards. That includes:  

• Clean, readable, and maintainable code 

• Tests that cover both happy paths and edge cases 

• Team review and agreement on the chosen solution 

• Compliance with approved designs and consistency with the design system 

• Meeting observability requirements 

• Deployment readiness, such as feature flags when needed and the ability to roll back safely 

For example, implementing a feature isn’t just about writing a component that works:

export const BetsTable = ({ bets }) => (
  <table>
    <tbody>
      {bets.map((bet) => (
        <tr key={bet.id}>
          <td>{bet.team}</td>
          <td>{bet.amount}</td>
        </tr>
      ))}
    </tbody>
  </table>
);

 A feature like this may technically satisfy the acceptance criteria, but is it "done"?

From a developer’s perspective, "done" might also require: 

• Compliance with approved designs 

• Handling of edge cases and non-happy path states, like error handling, empty data 

• Accessibility and semantic correctness 

• Consistency and usage of a design system 

export const BetsTable = ({ bets, loading, error }) => {
  if (loading) return <div role="status">Loading bets...</div>;
  if (error) return <div role="alert">Error loading bets: {error}</div>;
  if (!bets || bets.length === 0) return <div>No bets available.</div>;

  return (
    <TableWrapper
      data={bets}
      columns={[
        { header: "Team", accessorKey: "team" },
        { header: "Amount", accessorKey: "amount" },
      ]}
    />
  );
};

Another important factor in defining "done" is the collaboration with our QAs. They ensure that the feature meets the predefined requirements, while validating that the whole system functions the same way as it did before. The QA team brings a different perspective, challenging and exploring the feature beyond the happy path, asking and raising scenarios that developers might fail to notice. In fact, looking back, "done" work starts before the coding and testing processes. As a team, it’s crucial during refinement to ask about all possible outcomes of a given task. This way we can minimize unexpected scenarios and prevent bugs from appearing later in the process. 

Ultimately, Developers, QA, and Product each have their own perspective on what completion means. When "done" is not clearly defined, this often leads to misalignment and unnecessary rework. That is why it is essential for everyone involved to agree on a shared level of confidence, one that clearly outlines the requirements and expectations needed for a feature to behave as intended, while addressing the quality standards that matter to all roles. 

"Done" Belongs to Everyone

One of the easiest ways for a definition of "done" to fail is when it quietly becomes the responsibility of a single role. Sometimes it’s assumed to belong to Engineering Managers, Product Owners, Developers, or QAs. In practice, as in other aspects of the Software Development Lifecycle, such as Quality, the definition of "done" only works when it’s treated as a shared responsibility across the team.  

Through the Eyes of the Engineering Manager 

From an Engineering Manager’s perspective, shared ownership of "done" is less about control and more about creating the right conditions. 

That means encouraging early conversations about quality and risk during refinement and not just at the end of a story. It means making space for concerns to be raised without fear of being seen as blockers. And it means supporting teams in holding themselves accountable to the standards they agreed upon. 

Ultimately, when "done" belongs to everyone, the EM’s role shifts from enforcing a process to reinforcing a culture of responsibility, trust, and continuous improvement. 

Through the Eyes of the Developer 

From a Developer’s perspective, "done" is certainly not a personal milestone, it’s quite the opposite, a shared responsibility. This means involving everyone from the beginning of the definition so that the team can reach a shared level of confidence that the right conditions are met for a feature to be considered truly "done".

In summary, it's about collaborating early, asking the right questions, think of all possibilities, anticipate edge cases and aligning expectations across everyone involved. This way, we can ensure that we are able to deliver a feature with quality and stability.

"Done" is a Commitment, not Just a Checkbox

At its core, "done" is not a status update or a column in the Jira workflow. It’s a commitment the team makes to itself and to everyone who depends on its work. A commitment that the problem was understood, the solution was thoughtfully built, the risks were considered, the quality was evaluated, and the result can be trusted. 

For Engineering Managers, that commitment shows up as predictability, clarity, and trust in planning. For Developers and QAs, it shows up as confidence in the code, in the system, and in the decisions made along the way. 

When "done" is treated as a checkbox, responsibility becomes fragmented. Work moves forward, but questions remain unanswered. Edge cases surface late and quality becomes negotiable. From a developer’s perspective, this often means revisiting code that was considered finished. From a manager’s perspective, it leads to missed expectations and unreliable plans. 

But when "done" is treated as a shared commitment, something shifts. Conversations become more honest, trade-offs become explicit, and confidence in what is delivered grows naturally. All of these without the need for extra controls or last-minute validations. 

Reaching this point doesn’t require perfect processes or rigid definitions. It requires alignment, openness, and the willingness to revisit assumptions as teams and systems evolve. Most importantly, it requires recognizing that "done" is not owned by a single role, but rather by the collective. 

In the end, a strong definition of "done" reflects more than how a team works. It reflects what a team values: trust over shortcuts, clarity over assumptions, and responsibility over speed. When teams share that mindset, "done" stops being a risky word and becomes a reliable promise.

In our latest blog article, we explored the practice of dogfooding and how company-wide, in-house testing helps us sharpen product quality and ensure the best possible experience for our clients. But testing is only one side of the equation. If dogfooding strengthens what we already have, internal innovation pushes us toward what we haven’t built yet. Innovation and forward-thinking product design are just as central to our culture as quality assurance. They shape the way we imagine future features, improve existing systems and products and challenge ourselves to think beyond our current roadmap.

Last month, we took this commitment a step further by organizing a hacking competition, a multi-week initiative designed to bring people from different engineering teams and business areas together. The mission was simple, yet ambitious: collaborate, experiment and explore new ways to enhance our products, platforms, and/or processes. For 20 days, Blippers across backend, frontend, DevOps, QA, product and more disciplines dedicated focused time to building prototypes and demos, validating ideas and reimagining the future of our business.

Creating Blip Labs

Hackathons and short-term innovation events have been part of the tech playbook for years. Traditional 24-hour sprints or rapid design challenges are great for generating quick ideas and fast prototypes and they’ve produced meaningful results in the past. But for teams who still need to deliver on their day-to-day commitments, we wanted an internal model that was both sustainable and impactful, to generate innovative ideas.

We reimagined the format, creating an experience that preserves the excitement of a classic hackathon, while giving participants the time to dive into complex problems, validate architectural decisions and build thoughtful demos, all without disrupting the rhythm of the business.

And that’s how our hacking competition, Blip Labs, came to be. We supported Blippers in forming cross-functional teams, paired them with mentors to guide them, especially during the early ideation and architecture phases and then we stepped back. From there, it was up to our people’s brilliant minds to imagine, design and build.

What followed was three weeks of design sessions, coffee fuelled commits and spontaneous idea-sharing huddles and meetings.

Organizing the Competition

To prepare for Blip Labs, we first defined the competition stages and ensured that both the overall flow and the final event would be engaging for participants. We opened the application period and hosted an initial session to explain the format and introduce the participating teams and mentors. Throughout the competition, we held check-ins to assess progress and provide guidance. We set the submission deadline and organized a final ceremony, where teams presented their projects to the judges.

Along the way, we also created communication channels - including a dedicated Slack space - to keep participants informed about all competition-related updates. The channel was open to the entire company, not just the teams involved, because we wanted everyone to be part of the conversation, encourage participants and actively contribute to shaping new ideas.

Why a Multi-Week Format?

• Production-grade demos

Many of the final projects included fully functional demos, explanatory walkthrough videos and even the development of mobile applications, showcasing to the judges not just the concept, but the real potential of each idea.

• Cross-collaboration

Engineers, product managers, designers, DevOps specialists and QA engineers worked side by side with the organizing team, judges and mentors to ensure that even early-stage prototypes delivered real user value. This collaboration enabled meaningful usability testing, code reviews and architectural discussions, while still dedicating time to frontend polish, user experience design and product development.

• Culture of innovation

The extended format encouraged the habits we want our product culture to embody: experimentation, continuous testing and validation and a strong focus on the client experience throughout the product development process. It also gave us the opportunity to actively celebrate technical curiosity, pushing people out of their comfort zones and beyond their usual tools and frameworks, motivating them to explore new technologies and new ways of thinking.

How the Competition Worked

Teams were free to tackle any domain from new product development, enhancements to existing platforms, automation of internal processes, or architectural improvements. The only criteria were:

• The idea must have real business potential
• The idea must be functional, not hypothetical
• Collaboration should be at the center of the project/idea development

At the end, every team pitched their project to a panel of judges composed of engineering leads, directors and cross-functional experts.

Awarding the Winners

It wouldn’t be a competition without winners. After weeks of hard work, we closed Blip Labs with a final ceremony where all teams presented their ideas to a panel of jury members. It was a moment to celebrate the innovation Blippers brought to life and to recognize not only the overall winner, but also outstanding achievements across multiple award categories, each highlighting a different dimension of innovation and product development:

• Best Project (Overall Winner)

The Best Project was awarded to the idea that stood out the most in vision, execution, impact and presentation.

• Best Customer Experience

The Best Customer Experience recognized the project that delivered the most intuitive and thoughtful experience for clients. This category celebrated the solution that best understood clients’ needs and transformed them into engaging journeys.

• Best Technical Innovation

The Best Technical Innovation was awarded to the project that pushed the boundaries of engineering through creative architecture and problem-solving. It highlighted technical excellence that expands what’s possible within our product ecosystem.

• Best Pitch

The Best Pitch prize was awarded to the team that best communicated their vision with clarity, confidence and engagement. This category recognized members who made their solution compelling, easy to understand and inspiring to the public.

• Best Collaboration

The Best Collaboration Award celebrated the team that best represented cross-functional partnership. This award was given to Blippers who combined diverse skills and projects to bring new perspectives to their presentation.

• Blip Community Favorite

Chosen live during the event directly by the Blip community, this award reflected the project that captured the company’s enthusiasm and imagination. It represents creativity, relevance and the ability to resonate with people cheering on the sidelines.

Blip Labs: Where People Meet Innovation

After three weeks of brainstorming and building, it became clear that innovation flourishes when people are given the time and freedom to think boldly, either about our existing products and processes or about entirely new possibilities. The extended format, unlike the typical 24-hour events, enabled teams to architect with intention, integrate with real systems, test and refine their concepts into prototypes strong enough to spark conversations about actual deployment. Collaboration was just as essential: when backend, frontend, DevOps, design, QA and product representatives, with different seniorities and experiences came together, we saw smarter decisions, client-focused thinking, fewer bugs and errors and solutions that brought value to the business.

These weeks reaffirmed that innovation is not an isolated event but a cultural habit, one that values curiosity, encourages breaking routine and empowers Blippers to stretch into their full creative and technical potential. By bringing everyone into the conversation about what our future could look like, we not only surfaced powerful ideas but also deepened collective understanding of our market and business.

This hacking competition reshaped our sense of what internal innovation can achieve. The quality of the prototypes, architecture, frontend work and product development demonstrated real market potential, but even more importantly, it strengthened connection within our teams. The insights and energy generated throughout this experience will continue to shape what we build next.

We are certain this is only the beginning of future Blip Labs editions.

In the fast-paced world of software development, it’s tempting to push features out the door and rely solely on QA teams or early adopters to find the cracks. But we, at Blip, swear by a different approach: dogfooding. Eating our own dog food means using and testing products and services internally before releasing them into the world. It’s not just a quirky techy saying, it’s a cultural commitment to quality, knowledge and involvement. 

When our engineers, architects, designers, product managers and even executives rely on the same tools they’re building, bugs get spotted earlier, user experiences improve, and our teams gain a deeper understanding of our clients’ needs. But involving everyone in testing isn’t just about catching errors, it’s about fostering a shared sense of ownership and driving innovation. When we’re all engaged with what we’re developing, we gain a clearer understanding of how to improve, evolve, and create truly game-changing products.

Bridging the Gap Between Development and Delivery 

Delivering software is hard. Delivering software that every team trusts and deeply understands is even harder. That’s where dogfooding comes in. By running our own builds in-house and putting them in front of everyone, we create a feedback loop that automated tests can’t match. 

In growing companies like ours, it’s important that every team understands what they’re working toward. When we’re focused on day-to-day tasks, it’s easy to lose sight of the bigger picture, the impact of each line of code and how it shapes the final product. By constantly using our own platforms and experimenting with ideas still in development, we bridge the gap between the development and delivery stages. This approach gives Blippers the chance to directly influence the end product, enhance user experience, and ensure no bug goes undetected. It also deepens everyone’s understanding of our business and strategic goals, involving the entire team in the full product lifecycle - from concept to delivery.

How We Dogfood: Internal Competitions 

So how do we actually get a taste of our own food? How do we get everyone involved in enabling new products, giving people a deep, hands-on understanding of new developments and how they impact our apps? We run internal competitions to rigorously test our platforms across all brands, covering both cross-brand products and brand-specific developments. This approach turns every participant into an active contributor to the enablement and success of new product launches. 

Our VPN-protected staging environments replicate production at scale, using similar deployment processes and infrastructures as well as production-like test accounts for realistic, privacy-safe testing. Dedicated slack channels, onboarding tasks for new challenges, and daily leaderboards make the entire process engaging and transparent at every stage of the competition, transforming our internal teams into a high-value QA force. 

These internal competitions act as a strategic way to bring every Blipper closer to our products and see them from the client's perspective. In turn, this helps us catch errors early, shorten deployment cycles, and build products our teams are genuinely proud of. 

And perhaps just as importantly, the competitive format encourages collaboration. Team members form new groups, exchange ideas and work with colleagues they might not normally interact with, strengthening our company culture. At the end of the competition there are also, of course, prizes for the winners, which is always a plus.

Why Dogfooding Matters: A Wrap Up 

By now, we’ve seen that dogfooding isn’t just a testing method, it’s a mindset for us. Here are the key principles that make it work and why we keep adopting it: 

• Everyone’s a tester: From developers to executives, everyone contributes with valuable feedback that improves product features and quality. 

• Real-world conditions: Internal environments mirror production as closely as possible, ensuring Blippers’ user experience is the same as our clients will find in our platforms. 

• Knowledge and understanding: By using the tools we build every day, our teams gain a deeper knowledge of how each feature works, how it affects users and how their individual contributions shape the overall product. This firsthand experience builds stronger technical insight and empathy for the clients. 

• Early detection, faster fixes: Continuous internal use allows teams to identify issues long before clients do. 

• Culture of ownership: When everyone uses the product, there’s a shared sense of responsibility for its success. 

• Cross-team collaboration: Dogfooding encourages communication between departments that don’t often intersect, creating stronger, more connected teams. 

• Data-driven improvement: Internal testing provides real usage data that informs smarter design and development decisions. 

• Innovation through experience: Testing our products leads to fresh ideas for improvement that only come from hands-on familiarity.

The Technical Side of Dogfooding  

Behind the cultural shift lies a robust technical infrastructure that makes company-wide testing possible and secure.

1. Environments 

Depending on what we’re testing, we might spin up a protected staging environment or dive straight into production servers. The goal is always the same: to mirror what our clients see and feel. 

For experiments where more control or security is required, we use staging environments that run on machines only accessible internally via VPN. They run the production code with data identical to what’s used in production, but interactions with real client-facing services are kept to a minimum. 

For very specific situations, we can use the production applications to run competitions or other testing events. These events rely on controlled test accounts and close monitoring. 

2. Security and Access Control 

Whenever we touch production systems, security takes center stage. Our clients’ trust depends on it. Every initiative is carefully coordinated so that Blippers can explore and test our products, without putting anything at risk. 

Thus, when promoting these internal competitions, we need to make sure the Security and Internal Fraud teams are aware and alert for any suspicious activity. 

Test accounts used in production applications need to be properly flagged as such, approved and monitored by the Internal Fraud team to prevent any armful or bad behaviours. When dealing with real currency, there is also the need to disable all deposit and withdraw features, especially when using production environments.  

Finally, we make sure that these accounts are short lived and they are deleted as soon as they outlive the scope for which they were created.

3. Monitoring and Analytics 

As all modern application systems, we have a wide range of monitoring tools. Most of the outcomes and conclusions of our tests and experiments rely on dashboards and reports already created by our data analytics teams for the real-world clients. 

Because we’re working with the same data infrastructure during these internal events, we get to see exactly how our systems behave, where friction hides, and how people really move through the product.  

We pair those metrics with the human side of feedback. What did Blippers struggle with? What surprised them? What made them smile? These stories, combined with data, paint the full picture. 

Every dogfooding event becomes a feedback loop, a mix of metrics and lived experience that guides our next improvements. It’s a reminder that analytics isn’t just about numbers, it’s about listening to the story they’re telling, and acting on it before our clients ever need to. 

4. Feedback and Suggestions 

The best way to collect feedback is to make it easy for everyone to provide it. 

As a company best-practice, most or our products and initiatives have open Slack channels, where anyone can join and share suggestions on improvements or report issues with those products. These channels act as a catch-all and are long lived and separate from specific events. They are monitored and regularly advertised internally. 

When running a competition, we tend to take advantage of the closeness to the participants and the underlying context to explicitly collect feedback through forms and other call-to-action. This helps us focus the feedback into a more fine-grained detail. 

5. Running a Competition 

Motivation to win is a big driving factor in all initiatives and dogfooding is no exception. Having the mechanisms to allow Blippers to use our products internally isn’t the same as actual engagement, especially if we take into consideration that no direct value can be extracted from normal usage. 

To captivate engagement, we run internal competitions with a dedicated prize pool, custom leaderboards, and fun tasks to focus attention on specific features where feedback is more important. 

The tools used for these initiatives are parallel to the regular production environment, so they are custom made. 

The leaderboard typically serves as the main focal point of the competition. We share this information through a simple web application hosted on an internal URL. To ensure data security, given that the application requires access to account balance data, it is only accessible through the corporate VPN.  

Some special considerations need to be taken into account: for daily updates, we can subscribe to filtered internal reports from tools such as Google Analytics or Looker and automatically feed that data into the web application. If real-time data is required, we can use an authentication token to access the wallet service and retrieve the balance data. This token is monitored and will often be limited in access to make sure no real client data is accessed by accident. 

When it comes to enabling testing involving a large number of people, most of the effort to enable dogfooding is organizational, since the goal is to use the products as they are presented to the clients, almost as a black box usage. Most technical hurdles are related with security, access control and monitoring. Different challenges arise when running competitions, since there is a need to foster and keep user engagement by running parallel applications like the leaderboards.  

Learning Through Dogfooding and Improving the Competitions  

Dogfooding isn’t just a method, it’s part of our DNA at Blip. Since the very beginning, we have been organizing internal competitions to get everyone on board with ongoing developments and ideas for future ones. It keeps us curious and deeply connected to the products and services we deliver. When we encourage everyone to be a tester, we don’t just find bugs, we find opportunities. Opportunities to simplify, to enhance and to engage our clients with experiences that feel seamless because we’ve already walked in their shoes.  

Over time, we’ve refined these competitions to make them more dynamic and meaningful. By turning internal testing into a company-wide practice, we transform every release from a simple handoff to a shared achievement. Each iteration becomes a reflection of collective effort, pushing each product further. 

Ultimately, dogfooding reminds us why we build in the first place. It’s about a constant drive to improve and evolve. When everyone is part of the journey - from idea to implementation - innovation stops being a goal and becomes a natural outcome. 

And, of course, dogfooding isn’t the only way we test products, drive innovation and promote collaboration - so stay tuned for the next one! 

Hugging a cactus is a good analogy to describe the first assignment of the newly created Design Systems team at Blip.

It’s not that we dislike cactus, on the contrary. Some of our team members’ video call backgrounds are memorable for the sheer number of this spiky flora on display. Some even say, one of them have sprouted inspiration for big project codenames within the company.

However, no matter how much we indulge in cactus appreciation - just look at its strong stem, its various arms, flowers and even the patterns from the spines – our little remaining self-preservation instincts ensue we don’t want to touch them, let alone hug them. In the end we did more than embrace a cactus. Like the desert camel, we ate it whole. “How on earth do camels eat a cactus?” - you ask. Let’s see what National Geographic has to say about the matter: 

“…The camel's rotating chew distributes pressure from the cactus and the papillae slide the needles vertically down the throat. This way, the sharp ends don't poke the camel as it ingests them…” 

Oddly enough, this is somehow a good analogy to how the first big assignment of Blip’s Design System team was handled from the start. It will make sense further down this article, we promise. 

What is a Design System? 

Here’s a couple of good and simple explanations of what a Design System is, from industry experts Brad Frost and Nielsen Norman.

- "A design system as the official story of how your organisation designs and builds digital products." - Brad Frost

- "A design system is a living, complete set of standards intended to manage design at scale using reusable components and patterns." - NN Group

The root of our cactus 

Let’s start at the root level. Why are we writing this? Imagine you work for a digital product company (if you’re reading this, you probably do already) and you’re faced with a dilemma: You are a designer or developer working for a brand with an ageing product. Through each ticket, you realize it’s getting harder and harder to design or implement new features and designs. Your works keeps getting red taped because “legacy code” or “hard-coded” reasons. Customers keep complaining about app performance and you feel the product is hitting a technical ceiling. You keep hearing about the need to rebuild or refactor, but there’s never the budget or time, and the risk-averse business nature might fear losing customers, traditionally averse to change. 

What does this mean? 

Legacy code: Code written by others or under a previous language, architecture, methodology, or framework that pertains to the current project. Yet still in use. Legacy code might be hard to rewrite because of integrations. 

Hard-coded: Hardcoding is writing code that is not easily modified or reused. A hard coded information cannot be easily changed without changing the source code of the program itself.  

Imagine the brand you work for was just acquired by a larger umbrella group, Flutter. Sudently some of your competitors are now part of your family. Within this family, you notice your newly met sister brand has recently undergone your pain, and they came out on the other side with a brand new, state of the art tech stack, full customizable and dynamic content management, faster, sleeker, more efficient.

You’re faced with two options: go through the same epic, or borrow the technology, save time, effort and money. Seems like a no brainer, but as in life, there are always good and bad implications on every decision we make. We’ve asked our designers to lay down some pros and cons of this approach, using the true and tested SWOT analysis. 

What’s a SWOT analysis? 

SWOT stands for Strengths, Weaknesses, Opportunities, and Threats, and so a SWOT analysis is a technique for assessing these four aspects of your idea, project or business. 

Eventually, the path chosen by your product directors was to adopt the technology from the new sister brand. It’s important to understand that this approach, although seemingly logical, won’t apply to every industry. The betting digital products industry is a pretty standardized affair, because each move away from the status quo must be met with confidence in terms of user experience.

This is because it’s very common for a single user to have many different competitor apps installed at the same time, browsing for the best price on a specific match or race. There’s always room for different approaches, but it will probably never be an AirBnB/Booking situation, where the products do similar things but with distinct user journey experiences. Betting is closer to Uber/Bolt levels of similarity, where prices are often more important than the experience or customer support. 

Cactus spike - No previous culture of working with Design Systems 

One challenge we faced collectively as an organization was the previous lack of culture in working with a Design Systems team. Every new process takes time to be advocated and engrained in the day-to-day routines. Humans are creatures of habit after all. 

You might asking yourself: If the technological foundations for all brands become the same, the user experience is expected to be similar and the market is the same, what’s different between betting products then? In an industry where for better or worse, standardization and resource sharing is becoming a common practice, the differentiation will rely increasingly on value, best prices, loyalty programs and marketing tone of voice.

This is not a new concept. It’s been happening for decades in the automotive industry for instance. Take a VW Golf, an Audi A3, a Skoda Octavia, or a Seat Leon. These are all very popular automobiles in Europe, and, believe it or not, they all are the same exact car underneath. So, like in Flutter group, the platforms and the technology are becoming the same to benefit from cost synergies. However, each brand still must cater for a different demographic, regional focus, and “trim levels” if we want to keep the automobile analogy.

Bringing one more brand into this fold of shared design and technology stack is just the first trial. Our ultimate goal is to grow this Design System to serve all the brands within the huge betting conglomerate that is Flutter.  

What is a House of Brands and a Branded House? 

• A House of Brands is home to numerous brands, each independent of one another, and each with its own audience, marketing, look and feel. Ex: Meta (Facebook, Instagram, Whatsapp).
• A Branded House maintains the focus on a single, well-known and consistent brand across all its products. Ex: Google (Gmail, Google Maps, Google Chrome).

A Siamese Cactus 

Let’s start going up through the stem. We are yet to work on a project that follows the initial vision to the letter. At this point we believe such undertaking only exists in our idealistic minds. Designing digital products that exist in reality outside our shiny Figma canvas, is more often about knowing how to navigate and negotiate compromises within your circle of stakeholders (developers, product managers, other designers…), than it is to know where to push the pixels. 

We can say that the Betting industry is usually split into two main product offerings: Sports Betting and Online Casinos. Although most brands will operate on both sides, usually each one will have its own digital app or web environment. Code and design may or may not be shared or the product decisions might follow a different stream. The thing is, when working with digital product, usually nothing is created from scratch. There’s always some inheritance we have to deal with, whether it’s legacy code, third party integrations or simply a matter of circumstances. 

Cactus spike - Everything, everywhere all at once.  

The herculean task of migrating the design and tech stack from SkyBet to Betfair was given just an 18 months' timeframe for completion. Design, stakeholders and tech teams that barely knew each other now were suddenly mixed in an high stakes environment. Compounding the pressure with lack of Design System culture, the ever-present project scope tweaks made this a challenging project to coordinate. 

When the decision to adopt SkyBet into the Betfair tech and design stack was made, Flutter UKI had two main libraries to choose from. “The Wall” powered Betfair, while “Abacus” powered the Paddy Power brand. Although The Wall eventually was chosen due to the state-of-the-art dynamic content management system on the Sports Betting side, the truth is that Abacus was more mature in its adoption on the Online Casino sub-product. 

You can probably see where this is going. From the vision of creating a single unified Design System to hold multiple brands, we immediately had to compromise for a less-than-ideal siamese scenario of maintaining two parallel systems. There’s a quote from Lucas Vallim, Lead Product Designer at Revolut (at the time of writing) that might resonate with this situation: 

"Modern tech culture is all about the low-hanging fruits, the fast wins, the MVPs, and the most optimized user flow. And that’s all fine. But once you’re in MVP mode, you'll never really get out of it."

This doesn’t mean we won’t ever unify the systems in the future. We just want to show that a plan and a execution of a plan will always have divergences. Also, as we had the opportunity to learn from chatting with Louis Ouriach and his folks from Figma, sometimes two systems are better than one. 

Cactus spike - Four years of technical debt

"The Wall”, although state of the art, wasn’t built without its struggles. In a company without a previous culture of working with a Design System, there was a multi-year gap created between Design and Tech teams. From naming to component architecture, to tokens. The misalignment between teams was obvious and well known. A four year gap that had to be narrowed while onboarding the SkyBet brand. Because we like analogies: It’s like fixing a train while it keeps moving full steam ahead. 1001 cars long. 

Two Arms, for Now: Design Systems Grow — So Do Teams 

A Cactus can have many arms. More arms mean more water and nutrients needed to keep it healthy. Although sharing code and design might reduce the amount of manual pixel pushing, there is still a maintainer-to-product ratio you have to follow to make sure the system keeps manageable. Greater control means greater accountability. From two designers working each on the The Wall and Abacus before SkyBet came into the fold, we had to grow our team to nine designers and a full development team. Does 15 Design System maintainers seem a lot of people to oversee a population of six million users? 

Cactus spike – Evolving from a Product Designer to a Multi-Brand Designer

In the past, when asked to design a new feature, designers knew what brand it was for. They knew which color palette to use. The scope was narrow and specific. However, moving to a multi-brand environment will require every design to be made agnostically, as it will have the potential to be re-used across every brand in the group. 

One particular strenght of the cactus is that if the stem is solid, you can grow lots of arms from it. Same with a Design System. If you build strong shared foundation (or roots), you’ll be able to add new brands on top much more easily.  

Some common and well documented Design System foundations are:

• UI Kit (Design) with flexible and agnostic components: Unfortunately, when most people picture a Design System, the first thing they think of is the UI Kit. A User-Interface kit is usually a Design File (made within Figma, Sketch, Framer…) that stores non-coded, visual reference components consumed and interacted by Designers. It’s important to notice that the users won’t ever interact with these components in the live product. They are visual representations of what the code should look like, but not a working component by itself.
  
  
• Working components library (Code): This is the equivalent of the aforementioned UI Kit, but for developers. It usually contains snippet of code ready to be copied and implemented in a codebase.  
  
  
• Guidelines: Guidelines can be synonymous with documentation or instructions on how to use a component. It can also be the brand palettes of not only just colors, but corner-radius, shadow styles, spacings, typography, borders, sizing, opacities etc. In a multi-brand environment, designers might be asked to work on different brands one after the other, or even at the same time. This will help designers navigate each brand's intricacies. 
  
  
• A Design Tokens repository: A design token is a design decision written in code. It’s a universal language that every coding language should be able to understand. For a company as large as Flutter, the design tokens are probably the most important creative artifact in our assets. It’s the single document (or multiple if you want) that holds every decision, every color, every theme, every mode, every semantic rule ever created by designers and developers. Design tools come and go, each new one requiring a costly and time-consuming migration of designs. Tokens make us think of ownership. Who stores your Design? Is it your company, or is it your tool? We like to think that if you have your .json file (hopefully in a shared repository) , with all the Design tokens, Figma or any design tool you use could cease to exist tomorrow, and nothing would be lost.
   
• A governance rule book: Governance is a set of rules that will explain how all the moving parts communicate between each other. All the teams, the stakeholders, the tools, the systems. Even with the best tools and guidelines, without governance, everyone will work in anarchy.
   
• A reference hub (whether a website or internal tool): This is supposed to be the shop everyone goes to fill their needs. Whether it’s a component, update on a process, or latest decisions around any product matter. 

Design Tokens: Flowers That Keep Your System Blooming 

A Design System story like this wouldn’t be complete if we didn’t mention today’s hot topic in the Design community: Design Tokens. For us in the Flutter UKI Design System team, the single main artifact we’d run away with if our house was on fire, would be the tokens repository. They are the single source of truth. The locker of all decisions. They are the neural terminals that connect Design to every bit of visual code you can see on the product. 

From Developer Tool to Design System Essential

Design tokens have been around for a while now. We can trace their origins back to 2014, when they were first experimented with by Jina Anne and her team at Salesforce. However, until recently they’ve been just a tool used by developers to create constant variables, instead of having to manually input numeric values and hex codes all the time while coding. In recent years, with the help of tools like the Tokens Studio plugin or Figma Variables, the concept of a design token is becoming mainstream, regardless of format.

A Real-World Breakthrough: The Light Mode Challenge

We believe there is a proportionality between the size and complexity of one company and the need to adopt Design Tokens. At Flutter, one of the first true breakthroughs with Design Tokens was in 2022, when Betfair Sportsbook, powered by The Wall decided they wanted to provide the capability to their users to switch the color scheme to a light theme (kind of counterclockwise since most of the industry went from light to dark modes).  

The timeline was not friendly. The team would have just about 3 months to go from “what is a token?” to a customer firing up the app and saying, “oh look, Betfair has a light mode now!” The two options on the table were the same as always: either duplicate every single screen, component, environment you’ve ever done and recreate it with light colors, and live with an infinite maintenance overhead forever, or learn to extract all the potential design tokens have to offer. We thankfully followed the latter.  

No One-Size-Fits-All

It’s important to keep in mind that each organization needs might require a different token approach. Although the W3C (World wide web consortium) is working on defining universal standards for Design Tokens, the open nature of tokens will always be able to adapt to different contexts and needs. Tokens are usually separated into three main purposes: Core, Semantic and Component Tokens.

What is a Design Token? 

While there are more specific answers to this question, generally all industry thinkers will tend to agree that a Design Token is a small, reusable design decisiona that make up a design system's visual style. Although technically you can name a token whatever you want, it’s common to see them occupying one of three conceptual levels: 

Core Token: A Core/Base token is a raw value. Ex: The colour hex #FFB80C is converted into the core token “yellow”, or “yellow-10” if part of a color scale. 

Semantic Token: The semantic token will give meaning to a core token. Ex: “yellow” core token is linked to “primary color” semantic token. 

Component Token: Component tokens are usually optional, but are useful to allow that extra mile of freedom to design, as every component will gain independence (when needed) from the semantic tokens. Ex: “primary color” semantic token is linked to “primary button background color”. 

At Flutter, being a huge conglomerate of brands and products working in many different markets, it made sense for our team having a full 3-layer token system. Doesn’t mean your team will have the same needs. 

Did you know? 

In Australian, cactus is used as slang meaning "dead, useless, or broken." 

Our Australian friends at Sportsbet must have had a laugh when they were told their siblings at Flutter UKI were working on project broken. 

How to Take Care of Your Cactus

Buying your cactus and bringing it home is where the true responsibilities start. If you let it as it is without care, water and nutrients, it will probably not develop much, or worse, it will die. Same goes for design systems. Creating one button might be the birth of a Design System. It’s as simple as that. The challenge, harder than feeding a cactus, is teaching our organization that re-using our button has benefits. Advocating, teaching, and evangelizing. This is the hardest part we’re facing with deploying a Design System. 

1. Looking at a Design System as a long-term investment

Once Albert Einstein said that the 8th marvel of the world was a financial concept we know as compound interest: Having money grow exponentially and automatically over time, with minimal input from the funds owner. 

We believe Design Systems also follow a logic of compounding profits, or if you do it wrong, compounding (technical) debt. 

One advertising point usually shared by advocates is that the Design System speeds up the design and development roll out. It doesn't. Not at least linearly and instantly. Like compound interest, you must let all the small decisions and design solutions compound over time, and automatically they’ll start taking effect and avoid issues without requiring intensive manual input all the time. It’s very hard to convince project managers to give away small instant gains for a larger benefit somewhere in the future. So like compound interest, Design System has to be dealt with like a long-term investment. 

2. Be aware of timings 

Here’s a thought that perfectly pictures what our Design System at Flutter went through in the recent past: 

“There are periods in your career when no matter what you say, people just won’t 'get' systems. (…) There will be other times where you’ll have incredible advocates who not only 'get' systems but invest in you and your team." - Dan Mall. 

Prior to SkyBet adopting The Wall design system, the Betfair team tried to implement a design system for at least four years. Big project plannings were drafted, presentations were put together and relevant stakeholders consulted. One thing was missing though: a specific and time-sensitive business reason to invest. Suddenly, when SkyBet needed to be onboarded into The Wall tech and design repositories, it became perfectly reasonable to ring-fence 15 people (designers and developers) into an official Design System team. No matter hour hard we pushed before, nothing ever happened, not because our communication was ineffective, but for sheer bad timing. 

3. Sticking to the plan 

One thing we’re learning is that no matter how many clever process diagrams, guideline sheets, videoconferences around tips and tricks etc. All of this won’t penetrate the audience without persistence and consistent messaging. Persistence is essential if  habits need to be worked. We’ve been trying to get closer to designers through several ways: creating open channels for communication between teams, empowering designers with the ability to report bugs or missing features directly to a JIRA backlog. Ring-fencing a Design System designer and developer to focus full time on supporting designers and fixing bugs. However, probably the most ambitious initiative we’ve undertaken was the hosting of our own internal conference, codenamed CCE (Creating Cohesive Experiences). A big challenge of course, but and as popular knowledge says: the best way to learn is by teaching. 

Conclusion 

This article is not a guide, nor advice. It’s a gathering of insights and personal perspectives from the specific nature of the business we work in. As a team who focuses on Design Systems, we keep trying to converge our creative outputs into clear rules and patterns, conforming, reusing... But then Design is a naturally divergent creative area, so compromise is unavoidable. Humans are creatures of habit, so although it will sound cliché, if your Design System isn’t adopted by the people it’s trying to serve, it’s not a Design System, it’s a just a warehouse full of idle assets and promising ideas. 

References and further reads: 

• Product design is going down a weird path, but we can still save it - Lucas Vallim
• The State of UX, 2025 - Fabricio Teixeira, Caio Braga
• Why does everything look the same? - Isaac Fagerli  

As software grows and evolves, automated testing shifts from a luxury to an absolute necessity. Unit tests shine at ensuring each piece of the puzzle functions independently, but they can’t guarantee that the whole picture comes together perfectly. So, how do we make sure everything runs smoothly without manually testing every feature after every update? Enter integration tests — your secret weapon for making sure all components work in harmony, so you can deploy with confidence, knowing your application is rock-solid from end to end. 

Introduction

In a recent project, we built a REST API behind an API Gateway, using Kafka for event-driven communication, PostgreSQL for persistent storage, and AWS Lambda for asynchronous operations. Given the distributed nature of the architecture, we integrated Testcontainers into our testing strategy to ensure reliable, realistic integration testing. 

By spinning up real PostgreSQL and Kafka containers during tests, we were able to validate microservice interactions in an environment that closely mirrors production. These tests run automatically in our CI pipeline via GitHub Actions for every commit, eliminating the need for shared or preconfigured environments. This setup ensures fast, consistent feedback loops and higher confidence in service behavior. 

The practical impact has been clear: fewer bugs reach production, development and QA cycles are faster, and developers are more confident in the safety of their changes. Onboarding has also improved, as new team members can run complete test suites locally without manual setup. 

This approach reflects modern engineering trends - cloud-native development, shift-left testing, and self-contained environments - and positions us to build more resilient and scalable systems with greater efficiency. 

Why Use Testcontainers? 

The Problem: 

Running integration tests often means ensuring that the entire infrastructure — databases, message brokers, and other services — is not only up and running but also pre-configured to a specific state. However, when these resources are shared across multiple users or CI pipelines, test results can become unpredictable due to issues like data corruption and configuration drift. While some developers attempt to bypass these challenges with in-memory databases, embedded services, or mocks, these solutions introduce their own set of problems. In-memory services, for instance, may lack critical features or behaved differently from their production counterparts, leading to false positives and negatives in test outcomes.

The Solution: 

This is where Testcontainers comes into play. Testcontainers allows you to run your application’s dependencies, such as databases and message brokers, in isolated Docker containers, creating a consistent and reliable environment for your integration tests. By interacting with these real services instead of mock or in-memory alternatives, Testcontainers ensures that your tests are as close to the production environment as possible. This not only eliminates the unpredictability caused by shared resources but also provides a programmatic API that makes it easy to manage and control these containers within your test code. The result is more accurate, repeatable tests that give you confidence your application will behave as expected in the real world. 

Create a Simple Spring Boot Application 

Before we dive into writing integration tests with Testcontainers and Spring Boot, let’s start by building the application we’ll be testing. And where better to kick things off than at every Spring developer’s favorite playground — the Spring Initializr. 

For our example, we’ll keep things straightforward: a simple controller that connects to a repository and stores books in a PostgreSQL database. We’ll be using Java 21, Gradle with Groovy, and the latest Spring Boot 3.3.2 — keeping it fresh and modern as of this writing. 

To get our project up and running, let’s sprinkle in some magic with these must-have dependencies in your build.gradle file: 

implementation 'org.springframework.boot:spring-boot-starter' 
implementation 'org.springframework.boot:spring-boot-starter-data-jpa' 
implementation 'org.springframework.boot:spring-boot-starter-web' 
implementation "org.apache.httpcomponents.client5:httpclient5" 
 
runtimeOnly 'org.postgresql:postgresql' 
 
compileOnly 'org.projectlombok:lombok' 
annotationProcessor 'org.projectlombok:lombok' 
 
testCompileOnly 'org.projectlombok:lombok' 
testAnnotationProcessor 'org.projectlombok:lombok' 
 
testImplementation 'org.springframework.boot:spring-boot-starter-test' 
testImplementation 'org.springframework.boot:spring-boot-testcontainers' 
testImplementation 'org.testcontainers:junit-jupiter' 
testImplementation 'org.testcontainers:postgresql' 
testRuntimeOnly 'org.junit.platform:junit-platform-launcher'

Here’s how we’re going to lay out our project — think of it as the blueprint for building our book-storing application:  

. 
└── controller/  
│ ├── dto/  
│ │ └── BookDto.java // Book data transfer object 
│ ├── BookController.java // The Application entry point 
│ └── database  
│ ├── dao 
│ │ └── Book.java // The book database entity 
│ └── BookRepository.java // The book repository 
│ 
├── build.gradle // Project configuration and settings 
└── application.properties // Application properties configuration

We’ll kick things off with the Book class, which will serve as our database entity. It’s a simple yet essential structure, featuring an id, author, and title — just what we need to start storing our books. 

@Entity 
@NoArgsConstructor 
@AllArgsConstructor 
@Data 
@Builder 
public final class Book { 
 @Id 
 @GeneratedValue(strategy = GenerationType.AUTO) 
 private Long id; 
 @Column(nullable = false) 
 private String title; 
 @Column(nullable = false) 
 private String author; 
}

With the Book class in place, it’s time to create the BookRepository to handle our database interactions. We’ll keep it simple by extending JpaRepository, giving us a powerful and streamlined way to manage our book data. Here’s how it will look like: 

public interface BookRepository extends JpaRepository<Book, Long> { 
}

With our database classes in place, it’s time to configure our application. Add the following properties to your application.properties file to seamlessly integrate with our setup. For reference, the GitHub repo includes a Docker Compose file that sets up a PostgreSQL container with these connection properties: 

spring.application.name=book 
server.port=8080 
 
spring.datasource.url=jdbc:postgresql://localhost:5432/bookdb 
spring.datasource.username=bookUser 
spring.datasource.password=password 
 
spring.jpa.hibernate.ddl-auto=create-drop

Next up is our application’s entry point: the BookController. This will be a RestController with the /books request path. To keep things concise, I’ll show just the createBook method here, but you can check out my GitHub page for the full code. The controller is straightforward, with BookRepository as a dependency and basic validations, like checking if a book exists when fetching all books or ensuring the book has a title and author when creating a new one. Here’s how it looks: 

@Slf4j 
@RestController 
@RequestMapping("/books") 
@RequiredArgsConstructor 
final class BookController { 
 
 private final BookRepository bookRepository; 
 
 @PostMapping 
 ResponseEntity<Void> createBook(@RequestBody final BookDto bookDto) { 
 try { 
 
 if (!StringUtils.hasText(bookDto.getTitle()) || !StringUtils.hasText(bookDto.getAuthor())) { 
 log.warn("Invalid book data: {}", bookDto); 
 return ResponseEntity.badRequest().build(); 
 } 
 
 final var book = new Book(null, bookDto.getTitle(), bookDto.getAuthor()); 
 
 bookRepository.save(book); 
 log.info("Book created: {}", book); 
 
 return ResponseEntity.status(HttpStatus.CREATED).build(); 
 } catch (RuntimeException exception) { 
 log.error("Error creating book", exception); 
 return ResponseEntity.internalServerError().build(); 
 } 
 
 } 
 
}

As you might notice, the controller interacts with a BookDto instead of directly using the Book entity. This BookDto serves as a lightweight copy of the Book object, helping us keep the separation between the backend and frontend. Here’s what the BookDto looks like: 

@AllArgsConstructor 
@Getter 
@Setter 
@EqualsAndHashCode 
@JsonInclude(JsonInclude.Include.NON_NULL) 
public class BookDto { 
 private Long id; 
 private String title; 
 private String author; 
}

Writing Integration Tests with Testcontainers 

Now that we’ve got our application up and running, it’s time to take things to the next level with integration tests using Testcontainers. With the Testcontainers dependency already in place, we’re all set to dive into creating robust tests that will ensure our application runs smoothly. First, make sure Docker is running, as Testcontainers relies on it to create and manage containers at runtime. 

To kick things off, we’ll set up a custom PostgreSQL container. While it is possible to pull one directly from a Docker image, crafting custom classes allows us to have more control over our containers. We’ll create a singleton class that extends Testcontainers’ PostgreSQLContainer, giving us an adapted setup. Here’s what our custom container looks like:  

public final class BookPostgresqlContainer extends PostgreSQLContainer<BookPostgresqlContainer> { 
 
 private static final String IMAGE_VERSION = "postgres:latest"; 
 private static BookPostgresqlContainer container; 
 
 private BookPostgresqlContainer() { 
 super(IMAGE_VERSION); 
 } 
 
 public static BookPostgresqlContainer getInstance() { 
 if (container == null) { 
 container = new BookPostgresqlContainer(); 
 } 
 return container; 
 } 
 
 @Override 
 public void start() { 
 super.start(); 
 } 
 
}

With our container set up, we’re ready to dive into writing tests. Personally, I prefer to keep configuration separate from the test logic. So, we’ll start by creating a BaseIntegrationTest class to handle the setup, and then build out the CreateBookTest class that extends it. This approach keeps things clean and organized. 

The BaseIntegrationTest class serves as our foundation, running a PostgreSQL container with the power of @SpringBootTest. Before any tests kick off, we’ll set up Spring’s RestTemplate to perform REST requests to our Spring Boot application. This will allow us to interact with our API just like a real user would. Additionally, we’ll start the PostgreSQLContainer, which will serve as the database for our application during testing. A key player here is the @Testcontainers annotation, which ensures that all fields annotated with @Container are managed properly — taking care of the lifecycle of our PostgreSQL container. 

Next, we’ll configure our application properties to leverage the data from our container, including the username, password, and JDBC URL. This is done with the @DynamicPropertySource annotation from Spring, which allows us to dynamically inject these properties at runtime, ensuring that our application seamlessly connects to the containerized database during tests. 

This setup keeps everything running smoothly and ready for action. 

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT) 
@AutoConfigureMockMvc 
@DirtiesContext 
@Testcontainers 
public class BaseIntegrationTest { 
 
 @Container 
 protected static final PostgreSQLContainer<BookPostgresqlContainer> postgreSQLContainer = BookPostgresqlContainer.getInstance(); 
 protected static RestTemplate restTemplate; 
 @LocalServerPort 
 protected Integer port; 
 
 @DynamicPropertySource 
 static void registerDynamicProperties(final DynamicPropertyRegistry registry) { 
 registry.add("spring.datasource.url", postgreSQLContainer::getJdbcUrl); 
 registry.add("spring.datasource.username", postgreSQLContainer::getUsername); 
 registry.add("spring.datasource.password", postgreSQLContainer::getPassword); 
 registry.add("spring.datasource.driver-class-name", postgreSQLContainer::getDriverClassName); 
 } 
 
 @BeforeAll 
 protected static void setup() { 
 postgreSQLContainer.start(); 
 
 restTemplate = new RestTemplate(); 
 final HttpClient httpClient = HttpClientBuilder.create().build(); 
 final HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory(httpClient); 
 restTemplate.setRequestFactory(requestFactory); 
 } 
 
 @AfterAll 
 protected static void tearDown() { 
 postgreSQLContainer.stop(); 
 } 
 
}

For our create-book integration tests, it’s as easy as extending the BaseIntegrationTest class to set up your configuration. Then, simply use RestTemplate to perform the requests. This approach keeps everything streamlined and your tests looking sharp. Check out the example below to see how effortlessly you can create clean and effective tests. 

class CreateBookTest extends BaseIntegrationTest { 
 
 private static final String AUTHOR_NAME = "Author"; 
 private static final String BOOK_TITLE = "Title"; 
 
 @Autowired 
 private BookRepository bookRepository; 
 
 @Test 
 void createBookWithSuccess() { 
 final var bookDto = new BookDto(null, BOOK_TITLE, AUTHOR_NAME); 
 
 final var response = restTemplate.postForEntity( 
 String.format("http://localhost:%d/books", port), 
 bookDto, 
 Void.class); 
 
 assertEquals(HttpStatus.CREATED, response.getStatusCode()); 
 
 bookRepository.findById(1L).ifPresentOrElse(book -> assertAll( 
 () -> assertEquals(BOOK_TITLE, book.getTitle()), 
 () -> assertEquals(AUTHOR_NAME, book.getAuthor()) 
 ), () -> fail("Book not found")); 
 } 
 
 @Test 
 void errorCreatingBookNoTitle() { 
 final var bookDto = new BookDto(null, null, AUTHOR_NAME); 
 
 try { 
 
 restTemplate.postForEntity( 
 String.format("http://localhost:%d/books", port), 
 bookDto, 
 Void.class); 
 
 } catch (final HttpClientErrorException e) { 
 assertEquals(HttpStatus.BAD_REQUEST, e.getStatusCode()); 
 } 
 
 } 
 
 @Test 
 void errorCreatingBookNoAuthor() { 
 final var bookDto = new BookDto(null, BOOK_TITLE, null); 
 
 try { 
 
 restTemplate.postForEntity( 
 String.format("http://localhost:%d/books", port), 
 bookDto, 
 Void.class); 
 
 } catch (final HttpClientErrorException e) { 
 assertEquals(HttpStatus.BAD_REQUEST, e.getStatusCode()); 
 } 
 
 } 
 
}

Conclusions 

In this article, we’ve explored the crucial role of integration tests in ensuring our applications run smoothly from end to end. By leveraging Testcontainers, we’ve seen how to streamline the testing process, providing a clean and controlled environment for our tests. This approach not only simplifies managing dependencies but also enhances the reliability of our test results. With integration tests and Testcontainers in your toolkit, you can confidently deploy your applications, knowing that every component is carefully controlled and ready for action. 

To check out the complete codebase and see everything in action, visit the GitHub repository page. 

The following article is based on a presentation by Tiago Pinho, Blip's Head of Product, that took place at the Product Weekend event on February 28th and March 1st, powered by Blip, where key topics on Product Management were discussed, offering valuable insights and reflections on the matter.

Introduction

Experimentation is the key to continuous improvement, innovation, and data-driven decision-making in modern organizations. Yet, many large companies struggle to implement a structured experimentation culture due to resistance to change, misaligned KPIs, and lack of top-management support. By embedding experimentation into their core business processes, organizations can optimize products, improve customer experience, and ultimately drive higher revenue growth. This article explores how experimentation can transform businesses, the challenges it presents, and how to successfully implement a testing mindset.

In large organizations, decision-making is often influenced by internal biases, past experiences, or executive opinions rather than objective data. This leads to inefficiencies, increased costs, and missed opportunities for growth. Many organizations implement changes without validating their impact, leading to unintended negative consequences such as customer churn, reduced engagement, or declining revenues.

By adopting a structured experimentation framework, organizations can minimize risks associated with changes, optimize their offerings through data-backed insights, and uncover new opportunities for innovation. Instead of making assumptions, companies can use real-world evidence to guide their strategic initiatives. This shift not only reduces uncertainty but also fosters a culture of continuous learning and improvement.

Experimentation has been at the core of success for many industry leaders:

• Barack Obama's 2008 election campaign used A/B testing on its splash page, increasing the email sign-up rate by 40.6%, leading to an estimated $60M in additional donations. This demonstrated the power of data-driven decision-making in political campaigns.
• Google’s 41 shades of blue experiment helped them determine the most engaging link color, generating an estimated $200M in additional revenue by optimizing a seemingly minor UI element.
• com runs thousands of A/B tests simultaneously to optimize user experience and increase conversion rates. Their rigorous experimentation culture allows them to continuously improve customer engagement and revenue.
• Amazon’s one-click purchase feature was tested to reduce friction in the checkout process, leading to significant improvements in conversion rates and reduced cart abandonment.

These examples highlight how structured experimentation can have a direct and measurable impact on business outcomes, reinforcing the importance of a data-driven culture.

The Betfair journey

Betfair is one of the Flutter International brands and operates worlwide across more than 50 countries. Back in 2019, Betfair products were facing many challenges mainly related with the dated underlying technology which has led the decision to invest in rewriting all Betfair products from the scratch in a different language (React Native). As part of this massive project, a new CMS was built and configured,  based on principles of content reusability and scalability. Why? Given that Betfair operates worldwide with the vision of becoming the number one brand in key markets, a "one size fits all" approach is simply not an option. Each market comes with its own unique regulatory landscape, customer behavior and sports cultural nuances. Our product has been designed with flexibility at its core, allowing it to be adapted and optimized for local needs. This is where experimentation becomes invaluable. In the end, near to 90% of our products content and functionality could be managed through configuration, without any dependency of tech squads intervention which provided great flexibility for finding the optimal content setup. Naturally, such great flexibility led to the question: What if we integrate the CMS with an experimentation engine, enabling an endless number of content combinations that could be tested against each other to see which performs better?

Challenges

When we set out to implement an experimentation mindset in our organization, we knew we were aiming for something bold and transformative.

The biggest hurdle wasn’t technical—it was the mindset. The project we were working on operated under a feature factory model, where the priority was to ship new features as quickly as possible rather than validating their impact. Short-term gains were prioritized over long-term learning, and teams were more focused on delivery milestones than on measuring real business outcomes. Additionally, resistance to change was evident. Some stakeholders were skeptical, questioning whether experimentation was worth the effort or if it would just slow things down.

But we didn’t settle. Using spare capacity from a few sprints, we built a proof of concept (POC) to demonstrate the value of experimentation. Once we had results, we organized demos to showcase what was possible, highlighting the power of the tool that we have just created and could be further improved. These early wins helped shift some perspectives, but the real turning point came with a leadership change.

This experience reinforced a crucial lesson: cultural change takes time, persistence, and proof.

The implementation

Going from experimenting on nothing to experimenting on everything came with its own challenges. Suddenly, experimentation wasn’t the bottleneck—scaling it efficiently was.

Competing Experiments and Resource Conflicts

One of the first issues we faced was competing experiments. We wanted to test different initiatives at the same time, but without a clear prioritization system, we ran into resource conflicts, and we didn’t have a way of avoiding results contamination due to overlapping tests. This led to a difficult prioritization of experiments to be run sequentially, which significantly increased the time of market of the features being tested.

Lack of Structured Communication and Knowledge Sharing

Another major gap was the lack of a structured communication plan for experiment results. Teams were running tests, but insights weren’t being shared effectively across the organization. This led to duplication of efforts—different teams unknowingly experimenting on similar things or failing to build on past learnings. It became clear that we needed a repository of experiment results and a process to synthesize insights across teams.

Upskilling and Ensuring Stakeholder Understanding

We also faced an upskilling challenge. While enthusiasm for experimentation was high, many stakeholders—from product managers/owners to engineers —didn’t fully understand best practices, statistical significance, or how to properly interpret test results. We had to invest in training to ensure all involved parties fully understood the role they had to play in each part of the experimentation process.

Technical Debt and Post-Experiment Cleanup

On the technical side, tech debt became a real liability. Experimentation requires rapid iteration, but when an experiment is completed, there’s additional work to be done either to gold plate it with quality standards (in case of successful variants) or to remove the variants code.  We had to enforce a structured post-experiment cleanup process to prevent long-term issues like cluttered code blocks causing performance concerns.

Increased Time to Market Due to Experimentation

Another unintended consequence was the increased time to market. Running experiments takes time—defining hypotheses, setting up tests, collecting data, and analyzing results. While the long-term benefit was clear, stakeholders were frustrated with the perceived slowdown in feature delivery. This required aligning expectations and refining our process to strike the right balance between speed and data-driven decision-making.

Cross-Discipline Collaboration and Alignment

Lastly, cross-discipline articulation became a challenge. Experimentation requires close collaboration between product, engineering, analytics, content operations, marketing, and customer support. Without proper alignment, misunderstandings arose about test designs, success metrics, or operational impact. We had to create cross-functional experimentation rituals—regular syncs, shared documentation, and clearer ownership structures—to ensure that every test was executed smoothly.

The Experimentation process

Our experimentation process contains the following steps:

1. Identify Areas for Testing
2. Segmentation
3. Hypothesis Formulation
4. Prioritization
5. Build and Run
6. Result Analysis and Iterate

1. Identify Areas for Testing

One of the most valuable sources for experimentation ideas is the voice of the customer. While ideas can come from various places—competitor analysis, industry trends, internal brainstorming—nothing compares to direct insights from those who interact with our product daily. To harness this, we’ve built a robust system to monitor customer touchpoints across multiple channels, including CRM surveys, app store reviews, user testing, in-app feedback, Google Analytics, internal feedback, social media, and customer service operations. Compiling insights from diverse sources proved to be a challenge, as data was collected in multiple formats—emails, spreadsheets, powerpoints, dashboards, … To address this, we assigned a Product Manager/User Researcher to each VOC touchpoint so they centralize and standardize the data, transforming it into a unified language that could be systematically analyzed, compared, and prioritized in our experimentation backlog.

2. Segmentation

After identifying what to test, the next crucial step is determining who to test it with. Our business spans multiple jurisdictions, meaning our customer base is incredibly diverse, differing in demographics, culture, purchasing power, betting knowledge, and behaviors. Additionally, operating in highly regulated markets adds another layer of complexity, as local legislation requires us to adapt our product offerings accordingly. This makes jurisdiction the first and most fundamental level of segmentation. Beyond geography, it’s essential to define which customer profiles we want to target. Should an experiment be applied to all users, or only specific segments? Are we testing with high-value or low-value customers? Are we looking at new users, experienced bettors, or casual players? Do we need to differentiate by age groups, device type (iOS vs. Android), or engagement level? Effective segmentation ensures that experiments yield relevant, actionable insights while maximizing impact and compliance across diverse user groups.

3. Hypothesis Formulation

To ensure our experiments are structured and measurable, we follow a simple yet effective hypothesis framework:

• Theory: We believe that <CHANGE>
• Validation: For <TARGET> will achieve <BENEFIT>
• Objective: We'll be successful when we observe an increase in <PRIMARY METRIC>, without negatively impacting <GUARDRAIL METRIC>.

Primary metrics represent the main goal of the experiment, directly tied to the business outcome we aim to optimize. Guardrail metrics, on the other hand, act as safety checks to ensure there are no unintended negative consequences.

For example, if we hypothesize that reducing the number of mandatory fields in the registration form will improve conversion rates:

• Primary metric: Conversion rate – the percentage of users who complete registration after starting.
• Guardrail metric: Customer support contacts related to registration issues – ensuring that while more users complete the form, they don’t face confusion or errors that could lead to an increase in support tickets.

4. Prioritization

When (and When Not) to Experiment

To help prioritize experiments and avoid the "everything must be tested" mentality, we developed an internal framework that has proven to be effective. This framework evaluates potential experiments from two key perspectives: the confidence level in their success and the effort required to run the experiment compared to fully building the solution. Based on these factors, here’s how we approach prioritization:

Low Confidence + High Experimentation Cost

This is the worst-case scenario. It’s risky and expensive.

Recommendation:
Don’t proceed — unless you can either:

• Increase confidence (e.g., gather more insights, user feedback, or prototype results)
• Reduce effort (e.g., simplify the experiment setup with engineering)

Low Confidence + Low Experimentation Cost

These are great candidates for testing. The stakes are low, and even a failed test can yield valuable learnings.

Recommendation:
Proceed with the experiment. Use it as a discovery tool with minimal risk and potential high upside.

High Confidence + Low Experimentation Cost

If you're confident the change will have a positive impact, testing is still useful—especially to measure the actual uplift.

Recommendation:
Run the experiment.

• Make the code reusable, since it’s likely to be fully implemented later.
• Use the test to confirm assumptions and fine-tune before full rollout.

 High Confidence + High Experimentation Cost

At this point, experimentation may not be worth the overhead. If the cost to test is similar to full implementation, testing adds unnecessary friction.

Recommendation:
Skip the experiment.

• Instead, phase-rollout the full solution.
• Gradually introduce it to users while monitoring impact and guardrails.
• This allows validation without delaying value delivery.

By following this framework, we ensure that experimentation remains focused, efficient, and strategically aligned—maximizing value while avoiding waste.

5. Build and run

To build and run effective experiments, we must consider three key aspects: randomness of the population, traffic volume, and experiment duration.

a) Randomness of the Population

Ensuring a random allocation of users is crucial to eliminate bias and produce reliable results. When using a third-party experimentation platform, this process is typically handled automatically. However, when implementing an internal solution, it is essential to ensure that traffic is split truly at random. One way to verify this randomness is through an A/A test, where all test groups are shown the same variant. If the groups are truly random, their primary metric performance should be statistically identical. Any significant differences may indicate an issue with traffic allocation that needs to be addressed before running A/B tests.

b) Traffic Volume

The number of users included in an experiment directly impacts the statistical confidence of the results. To illustrate this, imagine flipping a coin. If you flip it only three times and get three heads, you can’t confidently conclude that the coin is biased. Even after ten flips, a 6/4 split might still be due to chance. So, how many flips—or in our case, how much traffic—do we need to reach a reliable conclusion?

The required traffic volume is determined by the following formula:

Where:

• n = required sample size (traffic volume per variation)
• p = baseline conversion rate (current performance of the primary metric)
• d = minimum detectable effect (expected improvement in the primary metric)
• Z = Z-score corresponding to the desired statistical significance level (e.g., 1.96 for 95% confidence)

This formula helps ensure that experiments have enough data to reach a statistically significant conclusion while avoiding underpowered tests that yield inconclusive results.

c) Experiment Duration

The duration of an experiment is directly tied to the daily traffic volume of the product. Simply put, the more daily traffic we have, the faster we can reach statistical significance. The duration can be calculated using the formula:

6. Result Analysis and Iterate

Aspects to Consider When Analyzing Results

Once an experiment has run its course, analyzing the results correctly and thoroughly is just as important as designing the test itself. A common pitfall is focusing solely on whether one variation "won" without digging deeper into why the results turned out the way they did. To extract meaningful and actionable insights, several key factors must be considered. 

a) Statistical Significance and Minimum Detectable Effect (MDE)

The first and most obvious question is: Did the primary metric change in a statistically significant way? A minor fluctuation in conversion rate might not mean much if it falls within the margin of error, so checking for statistical significance ensures that any observed impact is likely due to the change itself and not random chance. Additionally, it’s important to verify whether the minimum detectable effect (MDE) was reached—if the test was underpowered, a potential uplift might have gone unnoticed simply because there wasn’t enough traffic or time to capture it. 

b) Guardrail Metrics

Beyond the primary metric, guardrail metrics must be closely monitored. A test that improves conversion but increases churn, customer complaints, or operational costs might not be worth implementing. If any guardrail metrics show red flags, it’s a strong indicator that the change, despite its positive impact in one area, might have unintended negative consequences elsewhere. 

c) Segment Consistency

Another critical aspect is segment consistency. Results should be analyzed across different traffic sources, device types, jurisdictions, and customer profiles (e.g., new vs. returning users, high-value vs. low-value customers). If an experiment shows a strong uplift overall but performs poorly for a particular segment, a one-size-fits-all rollout may not be the best approach. Identifying these differences can help refine targeting strategies and even inspire follow-up experiments tailored to specific user groups. 

d) Accounting for External Factors

Finally, external factors must always be accounted for. Seasonality, major sales events, holidays, and external market changes can all skew results, making it seem like an experiment had a larger or smaller impact than it actually did. For example, a betting platform might see increased engagement during the World Cup, or an e-commerce store could experience natural conversion spikes on Black Friday—factors that must be separated from the experiment’s true effect.

By considering these factors, we ensure that the insights drawn from our experiments are not only accurate but also relevant and applicable to real-world decision-making. 

e) Next Steps After the Experiment

Once an experiment concludes, the work isn’t over—whether it was a success or not, next steps must be carefully considered to ensure that insights are properly applied, and the product remains clean and efficient.

If the Experiment Was Successful:

• Gold-plating the Solution: If an experiment was successful, the next priority is to gold-plate the solution—meaning it should be refined, optimized, and implemented with high-quality standards. In many cases, experiments are designed for speed rather than perfection, which can lead to certain shortcuts in best practices, such as hardcoded elements, inefficient logic, or missing scalability considerations. Before rolling out the winning variant at full scale, it’s crucial to ensure that the final implementation meets production-level standards, follows proper coding practices, and integrates seamlessly with existing systems. This step helps future-proof the solution and prevents technical debt from accumulating. 

If the Experiment Was Not Successful:

On the other hand, if an experiment did not yield positive results, it’s essential to understand why before simply discarding the idea. There are several possible reasons:

• Was the hypothesis incorrect? Perhaps the expected user behavior shift didn’t happen as predicted.
• Were the test variations flawed? Bugs or UX friction could have negatively impacted the outcome.
• Did external factors influence results? Market conditions, seasonality, or unforeseen trends might have played a role.

Regardless of the reason, any unused or experimental code should be cleaned up to prevent dead code from accumulating.  

f) Sharing Results Across the Organization

Finally, sharing results across the organization is a critical but often overlooked step. Experimentation is not just about individual tests—it’s about building a culture of learning. By documenting and communicating findings—whether positive or negative—teams can avoid duplicating efforts, identify patterns across different tests, and apply learnings to future initiatives. A centralized repository of experiment insights can be invaluable in shaping strategic decisions and refining testing approaches over time.

In short, experimentation doesn’t end when results are in—it ends when insights are applied, technical debt is managed, and the organization learns from every test.

Conclusion: The Power of Continuous Experimentation

Embracing a continuous experimentation mindset enables companies to make better decisions, innovate faster, and stay ahead of competitors. Organizations that prioritize testing gain deeper customer insights, mitigate risks, and optimize their product offerings with real-world data.

However, this shift requires strong top management support and a break from the feature factory mindset. Experimentation thrives when guided by customer insights, making the Voice of the Customer a key source of hypotheses. Systematically monitoring customer touchpoints ensures that testing efforts are aligned with real user needs.

For reliable results, experiments must be based on strong hypotheses with clear primary and guardrail metrics. Ensuring random testing groups, sufficient traffic volume, and proper duration is critical to avoiding biased or misleading conclusions.

Finally, all results matter—even those that don’t show improvements. Negative or neutral outcomes still provide valuable learnings, helping refine future tests. By fostering a culture of continuous learning and data-driven decision-making, companies set themselves up for long-term success.

This post explores practical strategies for addressing database inefficiencies in Django Admin, focusing on optimization techniques that directly target query performance without relying on caching as the primary solution.

Introduction

Django is one of the most popular web frameworks out there, enabling rapid development of secure and maintainable websites. It provides a robust set of tools to build web applications quickly, including an auto-generated admin interface for managing application data. While Django simplifies many aspects of web development, from database to templating abstraction, optimizing its performance—particularly Django Admin pages—can be a complex task and often presents unique challenges.

While caching is commonly suggested to improve performance, it quickly becomes apparent that this approach may not be the most effective in certain cases. In this scenario, the primary performance bottlenecks stemmed from inefficient database queries rather than issues with view generation or URL processing. Caching can be helpful, but applying custom rules for query caching introduces unnecessary complexity and risks masking deeper problems. 

How We Use Django 

At Blip, we are the maintainers of Surface Security, a security intelligence and automation platform we developed in-house for our security needs, that is written in Python Django. Since its inception, we wanted to spend our limited resources in working on ways to retrieve data from all platforms we could get access to and to get there faster, there was a design decision to use the Django Admin as the main interface of the application, since this would provide all features we needed to display, maintain and work with the data we threw at the platform.

Over time, the application grew in data complexity and size, and we started experiencing performance bottlenecks in the application, where some pages took too long to load, or even fail to load due to timeouts. In addition, our application performance monitoring tools were reporting excessive N+1 queries in many of those models that were experiencing issues.

The Django Admin interface is a powerful feature for developers, allowing easy management of data within web applications. However, as applications scale and handle larger datasets, the performance of the Django Admin can degrade. Optimizing this interface is crucial not just for developers but for end users who rely on quick access to data and administrative functionalities. 

Django provides a thoughtful and thorough page on optimizing Django applications. The keyword to any attempt to improve something in the software industry is stated at the beginning of that page: What are we optimizing for? 

As with any web application, performance bottlenecks may appear due to many reasons: from database issues, all the way to HTTP issues (templating, connectivity issues in our infrastructure, inefficient logic in our components, etc). The first step is to then understand where our main problem lies and start working through the issues. 

As mentioned in the introduction of the section, we were sure the gross of our performance issues lied on the database layer – reading heavy queryset, unoptimized queries. We later learned how some of these inefficiencies emerged from the Django Admin templates, which we will also mention in this post. 

Broad Applicability of Optimization Strategies 

While the base of our test subject is a Django application leveraging the Admin interface (which is not a common setup), the strategies and concepts discussed here are not limited to just the Admin – in fact, the same principles apply to regular Django applications, and to any application outside the Django realm.  

Whether you’re building a user-facing web app, an API, or another admin-like interface, improving database efficiency and reducing these bottlenecks can lead to significant performance gains across the entire application, saving potential infrastructure costs (vertical or horizontal scaling) and improve the user experience of your customers. 

The Solution: Focusing on Database Efficiency 

To address the performance bottlenecks in the Django Admin interface, the focus was placed primarily on optimizing database queries. While caching can be a useful tool in some scenarios, the key insight was that database inefficiencies were the main issue—applying caching would likely have masked the underlying problems and being an intensive data I/O application caching queries specifically would be a considerable challenge on its own without introducing issues to the end-users. Instead, the optimization process revolved around reducing unnecessary queries, query efficiency, and refining how data is loaded and displayed in the admin interface. 

The following summarizes the main suggestions and lessons learned during this process, all of which can be applied to any Django application facing similar challenges. 

Tip 1: Leverage django-debug-toolbar for Efficient Performance Analysis 

Regardless of whether you have access to more advanced Application Performance Monitoring (APM) tools, the django-debug-toolbar is an invaluable resource for performance optimization work. This tool enables detailed analysis of SQL queries, pinpointing bottlenecks, showing where the slowness is happening and goes in-depth so users can understand what function call is actually slowing down. It is not covered by this post, but it can even do wonders when working with templates too. The django-debug-toolbar provides ample information to help you begin optimizing your application with minimal friction. It’s also straightforward to set up and use. However, a crucial note: ensure that the toolbar configuration is not enabled in production environments. Exposing this tool in a live environment can lead to significant security problems, especially if your application handles sensitive data or is publicly accessible. 

During the optimization of several views within the Django Admin, the SQL tab is going to be our primary focus, as it provides the most relevant information for identifying and resolving slow queries. Overall, the django-debug-toolbar is an essential tool for anyone looking to improve the performance of their Django applications. 

Tip 2: select_related for all Foreign Keys 

A common trait of data-driven applications is to have data relationships represented by foreign keys. This is a basic primitive concept where there is a connection between table A and table B, typically in the form of an additional column with the ID of the object representing this relationship. 

In Django, this is defined using the ForeignKey model field type. In the example below, we establish a relationship between an Author and a Book. A book has (at least) one author.  

copyclass Book(models.Model): 
    name = models.CharField(...) 
    author = models.ForeignKey("app.Author", on_delete=models.CASCADE) 
 
class Author(models.Model): 
    name = models.TextField(...)

And to represent this model in our Django Admin view, we typically write this admin model:

# In your admin.py 
@admin.register(models.Book) 
class BookAdmin(admin.ModelAdmin): 
    list_display = ("name", "author__name") 
    readonly_fields = ("name", "author__name") 
    search_fields = ("name", "author__name")

However, a setup like the above where Django displays a list of books and their author's name, it will execute one additional query for each book to fetch the associated author. As the number of books grows—reaching hundreds of thousands, for example—this results in a significant performance degradation. The page load time can increase to several seconds due to the inefficient execution of O(n²) queries, ultimately putting a considerable amount of strain on the entire application. 

To mitigate this, there are two effective solutions. One is a straightforward approach, while the other offers more granular control: 

• list_select_related=True in the Admin model definition: This tells Django to automatically prefetch all foreign key relationships of the relevant fields, reducing the number of queries needed. 
• Explicitly specify which foreign keys to preload (e.g., list_select_related=("author",)): This allows you to selectively optimize the relationships you need, rather than loading all foreign keys. There is a performance tradeoff in this option too, where Django will have to search the model's metadata and add the foreignkey fields to the prefetch, whereas this function immediately specifies which models to add. 

Both methods result in an optimized query structure using an inner join, transforming the problem from O(n²) queries into a far more efficient O(n) operation. In many cases, this simple adjustment has significantly improved performance, effectively eliminating the N+1 query problem. 

Relevant references: 

• https://docs.djangoproject.com/en/5.0/ref/contrib/admin/#django.contrib.admin.ModelAdmin.list_select_related 
• https://docs.djangoproject.com/en/5.0/ref/models/querysets/#select-related 

Tip 3: prefetch_related all those ManyToMany 

Many-to-many relationships are commonly used as well to represent complex associations between data entities. These relationships provide greater flexibility in modeling real-world connections within applications. To illustrate this, let's revisit the example from the previous section and refine it further for improved accuracy. A book can have multiple authors, and a better way to represent this relationship would be to change the Book model to have a ManyToMany relationship with the Author.

copyclass Book(models.Model): 
    name = models.CharField(...) 
    author = models.ManyToMany("app.Author") 
 
class Author(models.Model): 
    name = models.TextField(...)

This change represents accurate as the real world more accurately and it's a far better and more scalable solution to represent the problem than, for instance, keeping the ForeignKey relationship and stuffing all Authors Names into the Name field. This will raise problems when searching for books by a given Author, for instance. 

This approach also opens opportunities for more sophisticated data visualizations, such as tracking the number of books an author has contributed to, regardless of whether they were the sole author or part of a collaboration. 

The Django Admin view will remain like the previous one, but with enhanced functionality:

copy# In your admin.py 
@admin.register(models.Book) 
class BookAdmin(admin.ModelAdmin): 
    list_display = ("name", "author__name") 
    readonly_fields = ("name", "author__name") 
    search_fields = ("name", "author__name")

In Django, ManyToMany fields create a secondary table that just olds the ID of object A and the ID of object B.  

Failing to optimize calls against these tables can result in O(n³) queries for each book—one query for the intermediate table that stores the relationship IDs, and another for the destination table (e.g., authors) to fetch the associated data. This inefficiency can quickly become a performance bottleneck as data and traffic increases. 

In this case, list_select_related will not work, as it is designed for ForeignKey and OneToOne relationships. Instead, prefetch_related should be used to prefetch related data from the defined models and perform the joining operation in Python, rather than relying on the database. 

Unlike with select_related, there is no ModelAdmin attribute to leverage for this optimization. Instead, we need to take a more customized approach by overriding the get_queryset method for the object instance. Since this method returns a queryset, we can add the necessary prefetch_related calls to ensure the data is efficiently retrieved.  

copy# In your admin.py 
@admin.register(models.Book) 
class BookAdmin(admin.ModelAdmin): 
    list_display = ("name", "author__name") 
    readonly_fields = ("name", "author__name") 
    search_fields = ("name", "author__name") 
 
    def get_queryset(self, request): 
        return super().get_queryset(request).prefetch_related("author")

What the code is doing is invoking the `super().get_queryset()` first, which runs the normal code from the framework, and returns a queryset. Remember that querysets are lazy-loaded – at this point in the code, there are no SQL commands being executed. We then add prefetch_related to the queryset and return this modification of the queryset. 

With this simple adjustment, the Django Admin query execution becomes more efficient, reducing the operation to O(n) again. It now performs a single additional query to fetch the author's table, and any further processing is handled in Python, ensuring minimal strain on the database. 

Relevant references: 

• https://docs.djangoproject.com/en/5.0/ref/models/querysets/#prefetch-related
• https://docs.djangoproject.com/en/4.2/ref/contrib/admin/#django.contrib.admin.ModelAdmin.get_queryset 

Tip 4: Use annotations instead of model attributes 

In many scenarios, we need to perform some computations on top of our data, like keeping track of how many books an author has published. Several approaches can be taken to implement this functionality: 

1. Maintain a field in the model that updates the count each time the model is saved, incrementing the counter when a new book is added. 
2. Create a method on the model to calculate the count when needed and call this method in the admin view. 
3. Leverage annotations to perform the count at the database level, allowing the database to handle the computation efficiently. 

For example, consider the following model with a book count field: 

copyclass Author(models.Model): 
    name = models.TextField(...) 
    book_count = models.IntegerField() 
 
    def save(self): 
       # Pretty tricky logic goes here to increment book_count, ensuring no duplicate books, we are only increasing this when added to  
       return super().save(...)

Option 1 involves manually updating the book_count field, which quickly becomes overly complex and difficult to maintain. Given its challenges, this approach is generally not recommended. Furthermore, this adds logic to a method that should not be used for this purpose.  What about when the object is modified using the update() method and therefore save signal is never called? You will lose the counter updates. 

Option 2 involves adding a callable in the Admin class field list_display, to calculate the count when each object is loaded: 

copy@admin.register(models.Author) 
class AuthorAdmin(admin.ModelAdmin): 
    list_display = ("name", "book_count") 
    readonly_fields = ("name", "book_count") 
    search_fields = ("name") 
 
    def book_count(self, obj): 
        return models.Author.objects.filter(author=obj).count() 
 
    book_count.short_description = "Published of books"

While this approach may seem straightforward, it introduces a performance concern. For each author in the list, another query is fired individually to count the books, putting significant stress on the database to load the changelist view. 

Option 3 is the correct approach, using Django's ORM capabilities more efficiently to perform the count directly within the database query, minimizing the number of queries and reducing the load on the application: 

copy@admin.register(models.Author) 
class AuthorAdmin(admin.ModelAdmin): 
    list_display = ("name", "book_count") 
    readonly_fields = ("name", "book_count") 
    search_fields = ("name") 
 
    def get_queryset(self, request): 
        return super().get_queryset(request).annotate(num_books=Count("books")) 
 
    def book_count(self, obj): 
        return obj.num_books 
 
    book_count.short_description = "Published books"

Here, get_queryset is overridden again to annotate it with a num_books field that counts the books for each author. This is a far more efficient approach, as the database handles the counting in a single query, reducing the number of queries from O(n²) to O(n). 

This solution not only improves performance, but also simplifies the code. Django intelligently handles the necessary ID filtering when querying related models, which improves maintainability and readability. In the end, this method not only boosts performance, but also makes the codebase cleaner and easier to manage. 

Relevant references: 

• https://docs.djangoproject.com/en/5.0/ref/contrib/admin/#django.contrib.admin.ModelAdmin.list_display 
• https://docs.djangoproject.com/en/5.0/ref/models/querysets/#count 

Tip 5: Proper use of indexes field attribute 

While the tips in this post are presented in no particular order, understanding and optimizing your models' relationships and querying patterns is crucial before diving into indexing. 

One of the final optimization technique to consider is the correct use of indexes. Previously, adding db_index=True to a model field was the standard approach, but Django has since moved toward a more explicit and detailed declaration of indexes through Meta options. 

By using the db_index attribute or the indexes Meta property, you can create an index on specific fields, significantly improving query speed. This should only be used on frequently queried fields. This is particularly beneficial for fields that are often used in filter conditions or join operations. However, it’s important to note that ForeignKey, ManyToMany, and fields included in UniqueConstraints do not require an explicit index definition, as Django handles indexing for these automatically.

Relevant references: 

• https://stackoverflow.com/questions/59596176/when-we-should-use-db-index-true-in-django
• https://docs.djangoproject.com/en/5.0/ref/models/options/#django.db.models.Options.indexes
• https://docs.djangoproject.com/en/5.0/ref/models/fields/#db-index 

Tip 6: Query results caching 

While this post has focused primarily on the SQL parts that can be optimized, caching is sometimes a solution to our problems. Repeated queries are a good candidate for a special type of caching possible in the out-of-the-box tools of Django – caching query results. This can significantly improve the performance of the code and is particularly useful for blocks that repeatedly execute the same query, even with variable parameters, which can be a common pattern in many Django applications. 

To solve this problem, we can use Django's built-in caching engine, or we can use a smart solution using Python's built-in functools.cache decorator. They work differently and serve different purposes however, so it's worth to go over each one of them:

Using functools

This approach requires no additional packages or setup. functools.cache has been part of Python's standard library since Python 3.9, so any recent Python/Django project can benefit from this implementation. 

This option is best suited for cases where you want to optimize a long standing I/O process, when a given query is repeatedly happening in that process. Let's assume there is a piece of logic in your application that performs a query inside a loop: 

def my_method(self): 
    for entry in MyModel.objects.all(): 
        another_entry = AnotherModel.objects.filter(entry=entry).first() 
        pass

The query within the for-loop can be cached. But perhaps it's not convenient to cache it in the entire website because the next time this runs the results may have changed. However, if we cache the queries' results when it's running, we may decrease the processing time considerably. 

Here's how to implement it: 

copy@functools.cache 
def getAnotherModel(entry): 
    return AnotherModel.objects.filter(entry=entry).first() 
 
 
def handle(self): 
    for entry in MyModel.objects.all(): 
        another_entry = getAnotherModel(entry) 
        pass

With this modification, for the duration of the custom command, the query results will be cached in memory. This means that subsequent calls to getAnotherModel(entry) will retrieve the cached result instead of performing the same query again, significantly reducing database load. 

While this example may seem trivial, many housekeeping commands and other processes involve repetitive queries with identical parameters. By caching the results, we unload the database servers and improve overall application performance. 

Relevant references: 

• https://docs.djangoproject.com/en/5.0/ref/django-admin/#running-management-commands-from-your-code 
• https://docs.python.org/3/library/functools.html#functools.cache 

Using Django Cache Framework 

This option can suit you if you want to cache queries across the whole project. Typically, this is the solution if there is a query that repeats itself across the whole project. The remaining of the chapter purposely skips setting up the caching service as that is a different discussion in itself. 

Django caching framework supports many primites but the one we are looking for accesses the low-level API to cache the queryset's results. 

from django.core.cache import cache 
from .models import Book 
 
def get_expensive_query(): 
    # Try to get the data from the cache 
    books = cache.get('expensive_query_key') 
    if not books: 
        # If not cached, perform the query and store it in the cache 
        books = Books.objects.filter(author__name="Zé Carlos") 
        cache.set('expensive_query_key', products, 300)  # Cache for 5 minutes 
    return books

The main advantage of this approach is that this would be available everywhere in the project. Your view could invoke cache.get("expensive_query_key") and get the cached results, as opposed to the previous solution which would only be available at runtime and for the lifetime of the execution. 

This approach also comes with its challenges: 

1. Caching strategy is required, as we do not want to cache everything, and we need to compromise on expiration times. 
2. We need ways to invalidate caches when the underlying data changes. 
3. Caching may not solve your problems, so it's worth comparing first if it is actually helping (go back to tip 1, with the Django toolbar, as there is a tab specifically for caching metrics.

Inlines: A Performance Bottleneck to Consider 

Inlines have been a notable challenge when optimizing the Django Admin interface. While they offer significant convenience, the performance trade-offs often outweigh their benefits, particularly in complex, data-heavy applications. Searching for solutions to inline performance issues reveals numerous discussions, many offering quick fixes that adjust querysets or apply other hacks. In our case, the added complexity of such solutions didn't justify the performance improvements, but depending on your specific application, they might still prove valuable. 

This leads to the broader consideration of custom SQL queries. When standard optimizations no longer suffice, or when you’re working with particularly large datasets, custom SQL can offer a more effective solution. In our case, working with a large-scale database, custom SQL queries have been a game changer, significantly improving performance. However, this comes with its own complexities and requires careful planning. 

While custom SQL is outside the scope of this article, it’s important to note that inlines—if used excessively—can become a performance bottleneck. If your application relies heavily on inlines, consider revisiting their usage and exploring alternatives that may better suit your performance needs. 

Relevant References: 

• https://stackoverflow.com/questions/29647418/django-admin-inline-select-related
• https://stackoverflow.com/questions/49411204/django-excessive-queries-with-inlines
• https://stackoverflow.com/questions/16305908/slow-performance-for-django-admin-inline
• https://stackoverflow.com/questions/559701/django-queries-made-repeat-inefficient?rq=4 

Conclusion 

In this article, we've discussed various optimization strategies that can help improve the performance of Django applications, particularly when working with large datasets or complex models. From reducing query counts using select_related and prefetch_related to leveraging django-debug-toolbar for detailed query analysis, these tips provide a practical foundation for tackling common performance issues. 

However, optimization is an ongoing process. The techniques presented here, such as proper indexing and query caching, can make a significant difference, but they need to be tailored to the specific needs of your application. Whether you’re dealing with the performance trade-offs of inlines, implementing custom SQL queries, or refining other parts of your system, the key is continuous evaluation and adaptation to ensure your application scales effectively without compromising performance. 

Join us as we explore our use case of exhaustive testing during the refactoring process. Hear directly from the team about the challenges faced, the solutions implemented, and how this approach restored clarity and functionality while shaping the development process.

Tackling Complexity in Software

In the realm of complex software systems, managing "complexity hotspots" is a critical challenge for development teams. These areas are marked by high complexity and essential functionality, often becoming increasingly difficult to maintain over time. In this article, we will explore the concept of exhaustive testing as a vital strategy for ensuring software reliability, using the refactoring of our QuoteEngine—a subsystem responsible for calculating bet prices for our Cashout product—as a key example. 

As our system evolved, the QuoteEngine transformed into a complexity hotspot, leading to prolonged implementation times for small changes and an unexpectedly high defect rate. We observed alarming signs of distress in the codebase, including lengthy classes, convoluted methods, and the notorious "shotgun surgery" effect, where minor updates required adjustments across multiple classes. 

Meet the Team: The Minds Behind This Strategy

Behind every successful testing strategy are the people who envision, implement, and refine it. Our journey into exhaustive testing began with Joana Ribeiro (QA Engineer) and João Costa (Senior Software Engineer), both key contributors to the Cashout product. While enhancing the QuoteEngine, they identified that implementing exhaustive tests could significantly improve validation processes. This realization came amidst the absence of documentation or prior guidance on the subject, prompting them to innovate and share their learnings. Their collaborative effort not only optimized our system but also led to the creation of reusable insights aimed at helping others facing similar challenges.

When Testing Every Input Isn’t Feasible

Exhaustive testing, or complete testing, aims to evaluate a system by running tests on all possible input combinations. In the context of the refactored QuoteEngine, the goal was to ensure it produced consistent results with the original system across every input. This method is crucial when dealing with numerous parameters, as manual testing becomes impractical. 

While exhaustive testing theoretically helps detect bugs by evaluating all potential input scenarios, it often proves impractical in real-world applications due to the sheer number of test cases, especially in complex systems. For example, a simple application with a 4-digit numeric input has 10,000 combinations. As more variables are involved, the number of possible test cases grows exponentially, requiring significant time and resources, often exceeding typical development cycles. 

To manage this complexity, we focused on key inputs and variables influencing the system's behavior, allowing us to benchmark numerical, boolean, and string values (e.g., "inplay/preplay" events) without testing every single possibility. While exhaustive testing is an ideal concept, its limitations underscore the need for effective strategies. Techniques like boundary value analysis, equivalence partitioning, and risk-based testing help testers concentrate on critical areas, ensuring thorough evaluation without the impracticalities of testing every scenario. 

Refactoring the QuoteEngine: Challenges and Risks

Given the centrality of the QuoteEngine to our service—where even a small bug could lead to substantial revenue losses and reputational damage—it became clear that we needed to address these issues. We initially attempted several approaches, including surgical code changes, extensive documentation, and additional unit tests, but none fully resolved the root problems. This led to the decision to perform a large-scale refactor of the subsystem to simplify the code structure and enhance maintainability. 

However, large-scale refactoring presents challenges, particularly in validating that no regressions have been introduced. The lack of unit test coverage and the system's complexity made thorough reviews difficult. Although thousands of component tests existed, they were too coarse-grained to guarantee proper handling of all edge cases. This prompted us to explore exhaustive testing as a solution to ensure that the refactor didn’t introduce unintended side effects. 

Why Exhaustive Testing Was the Right Choice

The decision to implement exhaustive testing was driven by the complexity of the QuoteEngine. The subsystem had 37 different parameters, each with a variety of potential values, creating an enormous list of possible input combinations. Manually testing all these combinations was impossible due to time constraints and the potential for human error. Exhaustive testing enabled us to run these tests automatically, ensuring that the output from the refactored QuoteEngine matched that of the original system for every input permutation.

How We Designed and Executed Exhaustive Testing

1. Identifying and Categorizing Input Parameters

We began the exhaustive testing process by identifying the input parameters for the PriceCalculator within the QuoteEngine. This involved a detailed analysis of all variables influencing the calculation. Boolean inputs (e.g., inplay and HorseRacing) were defined with their two possible states: YES/NO. For numerical inputs, we applied techniques such as Boundary Value Analysis (BVA) to ensure comprehensive coverage by focusing on edge cases where transitions between boundaries occur—common areas for defects. For instance, with a threshold field set to 0.02, where X ≤ 0.02triggers different logic, BVA helped us define scenarios like 0.019, 0.02, and 0.021 to validate the boundaries effectively. Similarly, string inputs were evaluated to include only those values that significantly impacted the calculation, while redundant ones were excluded. Ultimately, this meticulous process resulted in a table comprising 37 inputs and their respective values for exhaustive validation. 

2. Automating the Test Execution

Next, we set up a testing framework to conduct the exhaustive tests. The test framework loads the csv with the variable and respective values (table above), inflates the test objects with the specific combination of variable values and for each compares the output of the original and refactored QuoteEngine. 

public void runExhaustiveTestSuite() { 
    List<List<String>> records = new ArrayList<>(); 
    try (BufferedReader br = new BufferedReader( 
            new FileReader( 
                    new ClassPathResource("Exhaustive_Testing.csv").getFile()))) { 
        String line; 
        while ((line = br.readLine()) != null) { 
            String[] values = line.split(COMMA_DELIMITER); 
  
            List<String> params = Arrays.asList(values).subList(1, values.length); 
            if (!params.isEmpty()) { 
                records.add(params); 
            } 
        } 
    } catch (IOException e) { 
        e.printStackTrace(); 
    } 
  
  
    List<String> params = Arrays.asList(new String[records.size()]); 
    LOGGER.info("Running for {} inputs", records.size()); 
  
    generateAndExecuteScenarios(records, params); 
  
    LOGGER.info("End of Exhaustive test run"); 
}

The system processes input data from the table and runs it through both the original and the refactored code. It then compares the results and logs any inconsistencies between the two. Initially, the focus was on executing the tests, but now the system is verifying the output and logging any differences it detects.

public void runScenario(Bet bet, List<String> params) { 

    Result newResult  = newEngine.calculateTradeoutPrice(bet, params); 

    Result oldResult = oldEngine.calculateTradeoutPrice(bet, params); 

  

    if (!newResult.equals(oldResult)) { 

        LOGGER.error("Inconsistent scenario, old={}, new={}, bet={}", oldResult, newResult, bet); 

    } 

  

    long count = counter.incrementAndGet(); 

    if (count % 1_000_000 == 0) { 

        LOGGER.info("Executed scenario so far: {}M, with params: {}", count / 1_000_000, params); 

    }

Proving the Approach: Results and Validation 

In total, 290 237 644 800 tests were executed, one for each permutation of the input parameters. This extensive test set ran for 10h hours, after which no discrepancies were found between the original and refactored versions of the QuoteEngine. This validation gave us the confidence to deploy the refactored system into production, knowing it would maintain the same functionality as before. 

It’s important to note that we did not use this exhaustive test suite as part of our ongoing regression testing strategy. Instead, it served as a one-time validation tool during the large-scale refactor. After successful deployment, we returned our focus to unit and component tests to ensure ongoing code quality. 

Lessons from Exhaustive Testing

Exhaustive testing proved to be an essential tool in validating the refactor of our QuoteEngine subsystem. By systematically comparing the outputs of the original and refactored systems across all possible inputs, we were able to confidently deploy a cleaner, more maintainable version of the subsystem. This not only reduced the technical debt associated with the QuoteEngine but also provided the foundation for future improvements and bug fixes. 

In a world where complex hotspots and shotgun surgery can slow down development and increase the likelihood of defects, exhaustive testing offers a powerful approach to ensuring that large-scale refactors do not introduce regressions. Though resource-intensive, this method allowed us to confidently address a critical component of our system and ensure that it continues to function reliably, now and in the future. 

In this collaborative article, we spotlight the contributions of Anabela Carvalho and Bárbara Salgado, Senior QA Engineers at Blip, as they share their use case with Playwright Fixtures. We'll explore how this powerful feature helps streamline test setup and ensure consistency across different scenarios. Read on to learn about their experiences with the tool and to access a hands-on guide for effectively implementing Playwright Fixtures in your testing.

Introduction

Quality Assurance teams at Blip are constantly innovating to ensure the reliability and quality of our platforms, especially given the dynamic and unpredictable nature of the data we handle. Developing and maintaining reliable regression test suites is challenging, but the benefits far outweigh the effort. Our applications are continually evolving, and we must be creative in testing scenarios that reflect real-world situations. For example, in the sports betting industry, we often recreate complex scenarios like penalty shoot-outs in football, where each kick is either untaken, scored, or missed. While these events don't happen every match, we use mocked data to simulate them regularly, ensuring our system performs flawlessly. In this context, Playwright, a versatile browser automation framework, plays a crucial role in enhancing our test coverage.

Testing Dynamic Content with Playwright

Playwright is a versatile tool for automating browser interactions and tests across different browsers and devices. It’s an open-source framework that provides a straightforward API that simplifies scripting even for complex scenarios. Because of this, we think it’s an ideal choice for modern web development and testing environments.

Reliable Movie and Book Lists on a Library Platform

Let’s start with our test subject – a library website that provides lists of available movies and books. Our website is very simple: it features a landing homepage with a navigation bar containing three options: Home, Movies or Books. 

When the user visits one of the content pages, let’s say Books, either by clicking on the navbar item ‘Books’ or accessing /books, they will land on the Books page. Our website will then consult an API to retrieve whichever books are available and show them to the user. 

Here’s an example request and example response for the API: 
 
HTTP GET /api/books

{ 
    "books": [ 
      { 
        "isbn": "1234", 
        "title": "Book title", 
        "author": "Author name", 
        "publish_date": "2020-06-04T08:48:39.000Z", 
        "publisher": "Publisher name", 
      } 
    ] 
}

If by any chance there are no books available, the website will show a message indicating so. 

And if we get an error while fetching the API, we then show another screen: 

And that’s basically every screen we may encounter while navigating this website. 

Test Pyramid Approach

For the test coverage of our Library we’ll follow the concept of the test pyramid which advocates for a balanced approach to automated testing by structuring tests into three levels: 

1. Unit Tests: At the base, these are fast, isolated tests that cover individual components or functions. 
2. Integration Tests: In the middle, these tests verify the interaction between integrated components or systems. 
3. End-to-End (E2E) Tests: At the top, these tests validate the entire application flow from start to finish, mimicking real user scenarios but are slower and more brittle. 

The pyramid suggests having more unit tests, fewer integration tests, and even fewer E2E tests to ensure a robust, efficient testing strategy. For this experiment, we will only focus on E2E and, more importantly, integration tests – ensuring our website functions correctly by testing it independently from the API that supplies its data. 

Test scenarios 

• Scenario 1 - End-2-end test: visit books page

For this scenario we’ll access our Library website and navigate directly to the Books page as a real user would.

test('E2E – visit books page', async ({ page }) => { 
 
	// navigating to our books website 
	await page.goto('http://localhost:3000/books'); 
 	
	// creating some constants to interact with page elements 
	const header = page.getByTestId('header'); 
	const bookList = page.getByTestId('bookList'); 
 	
	// asserting on the page's title 
	await expect(header).toHaveText("Blip's Library"); 
 	
	// asserting that there's a book list visible 
	await expect(bookList).toBeVisible(); 
}); 

As you can see, we’re using Playwright’s <page.goto> to access our Books page url, then asserting on some static content from our books page. 

• Scenario 2 – Integration test (mocked data): No books available 

Now, if we want validate a scenario where the API returns no books, we can’t rely on end2end tests, as this is unlikely to happen frequently. We still need, however, to cover all our functional paths – and we do show a message for ‘no books available!’ as seen previously. We will use mocked data to generate this condition and guarantee our website functions as expected when this situation arises. 

By using playwright’s <page.route>, we’re be able to intercept the API call to /api/books and modify it to our needs. As we know the contract of this API, we’re able to mock its response with an empty array.

test('there are no books available', async ({ page }) => { 

  // before navigating to website, we're going to mock our API. 
  // body will return an empty books array 
  await page.route('http://localhost:3001/api/books', (route) => { 
      route.fulfill({ 
      body: JSON.stringify({ "books": [] }), 
    }); 
  }); 

  // now we can navigate to books page 
  await page.goto('http://localhost:3000/books'); 

  // creating some constants to interact with elements 
  const header = page.getByTestId('header'); 
  const bookList = page.getByTestId('bookList'); 
  const noBooksMessage = page.locator('.no-books') 

  // asserting on the page's title 
  await expect(header).toHaveText("Blip's Library"); 

  // assert on no books available message 
  await expect(noBooksMessage).toHaveText('No books available!') 
 
});

This way, we can guarantee that we will always get this scenario covered and can now add it with confidence to our regression test battery.

• Scenario 3 - Integration test (mocked data): 2 books available 

Now, since in our initial end-2-end test couldn’t validate the actual content of a book because the API response can’t be predicted, we will use mocked data to present two books and assert its contents. 

As we did in the last scenario, using Playwright’s <page.route>, we’ll intercept the browser call to /api/books and modify it to our needs – we’ll create a valid response with two fake books.

test('there are 2 books available', async ({ page }) => {

  // before navigating to website, we're going to mock our API. 
  // body will return two fake books 

  await page.route('http://localhost:3001/api/books', (route) => {
    route.fulfill({
      body: JSON.stringify({
        "books": [{
          "isbn": "123",
          "title": "The Automation Blueprint",
          "subTitle": "Mastering QA in the Age of Continuous Delivery",
          "author": "Barbara Salgado",
          "publish_date": "2020-06-04T08:48:39.000Z",
          "publisher": "Blip QAs Editions",
          "description": "a book about automation in the QA world"
        },
        {
          "isbn": "124",
          "title": "Breaking the Flaky Test Cycle",
          "subTitle": "A Pragmatic Guide to Reliable Test Automation",
          "author": "Anabela Carvalho",
          "publish_date": "2020-06-04T08:48:39.000Z",
          "publisher": "Blip QAs Editions",
          "description": "a book about handling flaky tests"
        }]
      }),
    });
  });

  // now we can navigate to books page 
  await page.goto('http://localhost:3000/books');

  // creating some constants to interact with page elements 
  const bookList = page.getByTestId('bookList');

  // asserting book list is visible and taking a screenshot of its contents 
  await expect(bookList).toBeVisible();
  await bookList.screenshot({ path: 'booklist-2-results.png' });
});

In this example we are even able to cover this functional path with visual tests, because we are making sure that the results are always the same. This is what our test would find while running: 

• Scenario 4 - Integration test (mocked data): Error fetching API 

For this scenario we will once again use Playwright’s <page.route> to intercept the browser’s API call to /api/books but this time we won’t care about the response, we just want a status code 500. This way we can prompt the website to react and adapt its visual content to this edge case.

test('there was an error fetching the API', async ({ page }) => {

  // before navigating to website, we're going to mock our API. 
  // body will return an empty books array 
  await page.route('http://localhost:3001/api/books', (route) => {
    route.fulfill({
      status: 500,
    });
  });

  // now we can navigate to books page 
  await page.goto('http://localhost:3000/books');

  // creating some constants to interact with elements 
  const header = page.getByTestId('header');
  const bookList = page.getByTestId('bookList');
  const errorMessage = page.locator('.error')

  // asserting on the page's title 
  await expect(header).toHaveText("Blip's Library");

  // assert on no books available message 
  await expect(errorMessage).toHaveText('Error fetching API!')

}); 

Creating Fixtures

In Playwright, fixtures are a powerful way to set up the environment or context for your tests, allowing you to share setup and teardown code across multiple tests. They help ensure that each test has the necessary context and resources, and they promote code reusability by allowing shared setup and teardown logic. Looking at our previous test examples, every single test has a common shared setup: every test will instanciate some elements and every test will open the same url. This shared setup is a good example where a Fixture can be useful. The purpose of this new Fixture will be the following:

• Load page objects associated with this page, so we can easily interact with elements in our tests.
• Open the respective page by doing a <page.goto> to the correct URL.

import { Page, test as base } from '@playwright/test';

import { BooksPO } from '../page-objects/books.po';

// our new type of fixtures is called PageFixtures 
// for now we only have one Fixture, BooksPage 
export type PageFixtures = {
  BooksPage: BooksPO;
};

// extending playwright's basic test by providing a BooksPage Fixture 
const test = base.extend({
  BooksPage: async ({ page }, use) => {
    const booksPagePO = new BooksPO(page);
    await booksPagePO.openPage();
    await use(booksPagePO);
  },

export { test };

As you can see, for this  <PageFixtures> we extend Playwright’s Test and by doing so, we’re creating our own context of what our test can have access to. We now know that if use a BooksPage Fixture we’ll have this setup preloaded. At the beginning of each test, we’ll have the page ready to start validations. 

Let’s see how our E2E test will look like when we use this new fixture:

• Scenario 1 – End-to-End test: Visit books page (using fixtures) 

import { expect } from '@playwright/test'; 
 
import { test } from '../fixtures/page.fixture'; 
 
    // start a test using BooksPage fixture as a preloaded setup 
    test('E2E - visit books page, async ({ BooksPage, page }) => { 
   
        // using header and welcomeMessage from BooksPage PO 
        await expect(BooksPage.header).toHaveText("Blip's Library"); 
        await expect(BooksPage.welcomeMessage).toHaveText("Welcome to Blip's library!"); 
    });

As you can see, we created a lot less code this time, as BooksPage fixture is already encapsulating the navigation to the page and the declaration of the needed elements. Notching up the complexity, let’s see where we can apply Fixtures to our integration tests. 

Looking at our integration tests examples we can see that in the beginning of each test we use page.route to simulate different data: 

• An empty book list 
• A book list with 2 books 
• An error  

Applying the same concept as we did for BooksPage before, we can create new type of Fixture that preloads the needed mocked data as a setup for our test.

import { Page } from '@playwright/test'; 
// importing test from our previous state 
// test will now be extended with both BooksPage and these new Fixtures 
// allowing us to use both Fixture types in the same test 
import { test as base } from '../fixtures/page.fixture'; 
 
// adding a new type of Fixtures, BooksMockFixture with three new Fixtures 
export type BooksMockFixtures = { 
    WhenIHaveTwoBooks: any; 
    WhenIHaveNoBooks: any; 
    WhenThereIsAnError: any; 
}; 
 
// extending basic test by providing a new Fixture for each different simulated scenario 
const test = base.extend({ 
    WhenIHaveTwoBooks: async ({ page }, use) => { 
        await page.route('**/api/books', (route) => { 
            route.fulfill({ 
              body: JSON.stringify( 
                { 
                  "books": [ 
                    { 
                      "isbn": "123", 
                      "title": "The Automation Blueprint", 
                      "author": "Barbara Salgado", 
                      "publisher": "Blip QAs Editions" 
                      }, 
                      { 
                      "isbn": "124", 
                      "title": "Breaking the Flaky Test Cycle", 
                      "author": "Anabela Carvalho", 
                      "publisher": "Blip QAs Editions" 
                      } 
 
                  ] 
                } 
              ), 
            }); 
        }); 
        await use(page); 
    }, 
    WhenIHaveNoBooks: async ({ page }, use) => { 
        await page.route('**/api/books', (route) => { 
            route.fulfill({ 
              body: JSON.stringify({ "books": [] }), 
            }); 
        }); 
        await use(page); 
    }, 
    WhenThereIsAnError: async ({ page }, use) => { 
        await page.route('**/api/books', (route) => { 
            route.fulfill({ 
              status: 500, 
            }); 
        }); 
        await use(page); 
    }, 
}); 
 
export { test }; 

Now we’re able to update our three remaining integration test scenarios with our both BooksPage fixture and BooksMockFixtures. 

• Scenario 2 – Integration test: no books (using fixtures)

// the order of the parameter Fixtures is important 
// first we want to load the mocked data through page.route 
// then we want to open the page 
test('there are no books available', async ({ WhenIHaveNoBooks, BooksPage }) => { 
   await expect(BooksPage.noBooksMessage).toHaveText('No books available!') 
}); 

• Scenario 3 – Integration test: 2 books available (using fixtures)

test('I should see 2 book results', async ({ WhenIHaveTwoBooks, BooksPage }) => { 
	await expect(BooksPage.booksList).toBeVisible(); 
	await BooksPage.booksList.screenshot({ path: 'booklist-2results.png' }); 
}); 

• Scenario 4 – Integration test: Error fetching API (using fixtures)

test('I should see error message', async ({ WhenThereIsAnError, BooksPage }) => { 
	await expect(BooksPage.errorMessage).toHaveText('Error fetching API!'); 
}); 

As you can see, our tests are now solely focused on expectations as our Fixtures are doing all the setup for us. Because we’re being descriptive in titles and naming, the test remains readable (we still could improve readability using test steps). 
 

Conclusion 

Fixtures encapsulate setup and teardown in one place, are reusable across test files, and are initialised only when needed. They are composable, allowing complex behaviors through dependencies, and flexible, enabling precise environments without impacting other tests. Fixtures also simplify test grouping by eliminating the need for setup in describes. Nonetheless, Fixtures can introduce some drawbacks that might make them a less-than-ideal solution in certain contexts – they can introduce complexity, making tests harder to understand and manage, especially for new developers, by hiding setup logic and tightly coupling tests to specific configurations. When tests fail, debugging can be harder when fixtures are involved, since the failure might originate from fixture setup/teardown and tracking down the root cause of issues can take longer. Fixtures can provide powerful advantages in test organidation and flexibility, but in our opinion we should use them wisely to avoid added complexity, debugging challenges, and unnecessary overhead in simpler scenarios. 

This article explores our preparations and strategic initiatives during major sports events, focusing on how we ensure peak performance and customer satisfaction. We will discuss the meticulous planning behind our technical operations, highlight key initiatives such as optimising platform performance and decoupling cashout options, and share insights from our teams on managing high-demand periods of the Spring Racing season, that includes Cheltenham and Grand National. 

Why Event Readiness and Strategic Initiatives Matter 

In the dynamic landscape of technical operations, preparing for major events is essential to ensure our systems operate flawlessly under high demand. Months before events like Cheltenham and the Grand National, our technical teams meticulously analyse past metrics to identify areas for improvement and optimise our platform's performance. 

One significant initiative was the decoupling of cashout features from the Bet Reporting Flow during peak sports events. This decision aimed to streamline our architecture, reduce infrastructure costs, and enhance cashout success rates. By evaluating the impact of these changes during critical events, we aimed to achieve efficiency and improve customer satisfaction. 

Throughout these preparations, our teams work in war rooms, actively monitoring platform performance and swiftly addressing any issues. This proactive approach not only ensures smooth event operations but also highlights our commitment to delivering exceptional technical solutions that meet the rigorous demands of our industry. 

Project Overview: Key Facts, Figures, and Milestones 

In the rapidly evolving field of data processing, ensuring the efficiency and reliability of a system that handles large volumes of data is crucial. Our tech project, involving 12 Sportsbook (SBK) Platforms teams, 24 engineering managers and tech leads, and 71 engineers actively monitoring, focuses on optimising the processing and comparison of betting information across multiple data streams. Preparation for this project began on January 9th with the leadership team, followed by the first load test session on January 19th. Key collaboration dates included meetings with tech leads and service lifecycle managers throughout January. The project targeted major events like Cheltenham (March 12th to March 15th) and the Grand National (April 13th), requiring extensive coordination. The leadership team, engineering managers, and tech leads participated in 12 sync sessions, while 10 additional sessions involved service lifecycle managers. 8 overall load test sessions and 10 flow-specific load test sessions were conducted. During event days, over 200 operational tasks were performed early each day by the 12 teams, with operational checks before each race, ensuring smooth and efficient handling of the data flow. We will explore further the core methods that drive our system, detailing their functions and interactions. 

Case Study: Optimising Performance During Major Sports Events 

The bet information is continuously updated and published across multiple data streams. The bet reporting flow consumes these streams and stores the updated bet information. Previously, the cashout flow had to consult the bet reporting flow to obtain bet information. With this new project, the cashout flow now directly consumes information from the data streams. 

This change brought several advantages. Firstly, it completely isolated the bet reporting and cashout flows from each other. Secondly, the load on the bet reporting flow was significantly reduced. Thirdly, the cashout flow became more efficient and faster, gaining the capacity to handle more requests more frequently. This improvement allowed us to increase the success rate of cashouts for our customers. 

Overall, these methods work in concert to ensure that our tech project operates smoothly, efficiently, and reliably, handling large volumes of data with precision and speed. By leveraging parallel processing, we are able to maintain a high standard of data integrity and operational performance. Each method contributes to the robustness of our system, making it capable of meeting the demands of a dynamic data environment. 

General Operation Cycle of the Consumer 

With this project, cashout now consumes multiple data streams of bet information. These streams can send thousands of events per second, so assuring that we have a high-performant and robust consumer for them was vital. With this project, cashout now consumes multiple data streams of bet information. These streams can send thousands of events per second, so assuring that we have a high-performant and robust consumer for them was vital. 

The general operation cycle of the consumer is encapsulated within the run method. The run method is the "base" method of the consumer, meaning all the consumer's logic is contained within it. What this method does is it starts the consumer (handle startup), and then cyclically processes messages from the stream (converting them and storing them in an internal database) and commits the messages to signal that they have been processed. Eventually, when the consumer is stopped, the method handles the shutdown gracefully (handle shutdown). This method is crucial as it orchestrates the core activities of the consumer, ensuring it continuously processes incoming data. It sets up the necessary configurations, establishes connections, and manages the flow of data through the system. The run method ensures that the consumer remains operational and responsive to new data. 

public void run() { try { handleStartup(); while (true) { ConsumerRecords<K, V> consumerRecords = kafkaConsumer.poll(Duration.ofMillis(100)); if (!consumerRecords.isEmpty()) { processRecords(consumerRecords); processCommit(); } checkHealth(); } } catch (Exception exception) { handleShutdown(); KafkaConsumerExceptionHandler.handleKafkaConsumerException(exception, kafkaConsumer); } }

Parallel Processing of Records by the Processor 

Parallel processing of records is a significant feature of our system, managed by the processRecords method. This method enables the processor to handle multiple records simultaneously, optimising efficiency and performance. By processing records in parallel, we significantly reduce the time required to handle large volumes of data, making our system more scalable and responsive. 

protected void processRecords(ConsumerRecords<K, V> consumerRecords) { updateOffsetsTracker(consumerRecords); consumerRecords.forEach(consumerRecord -> Optional.ofNullable(shardedExecutor.getExecutorByKey(consumerRecord.key())) .filter(executorService -> !executorService.isShutdown()) .ifPresent(executorService -> executorService.submit(() -> { try { eventProcessor.processEvent(consumerRecord); kafkaConsumerOffsetsTracker.remove(consumerRecord.offset()); } catch (Exception exception) { thrownException.set(exception); } } ) ) ); }

Converting and Persisting Bets to the Database 

When it comes to data persistence, particularly the conversion and storage of betting information, the persistBet method is employed. This method is responsible for converting bet data into a suitable format and then persisting it into our database. Ensuring the accuracy and integrity of this data is critical, as it underpins many of our analytical and operational processes. 

private void persistBet(ConsumerRecord<String, BetDomainEventOuterClass.BetDomainEvent> consumerRecord) { try { Bet convertedBet = convertBet(consumerRecord.value()); retryableOperationHandler.handleOperationWithRetries(() -> dataStoreClient.persistBet(consumerRecord.value().getBet().getBetId(), convertedBet, getStreamCreationTime(consumerRecord.value()).orElseThrow()).join()); } catch (WakeupException | MaxRetriesExceededException exception) { throw exception; } catch (Exception exception) { handleException(exception, consumerRecord, PERSISTING_ACTION); } }

Bets Comparison Between Two Streams 

The cashout process used to obtain bet information from the bet reporting flow, but it now connects directly to the streams that send the bet information. To ensure we didn't introduce any bugs, we needed a mechanism to guarantee that a given bet, after being transformed into the internal cashout format, was the same whether it was consumed directly from the bet stream or obtained from the bet reporting flow. 

This method served to compare the bets and log any discrepancies. Since our logs did not record any differences, we concluded that there were no errors in consuming and converting the information from the bet streams. This gave us the confidence needed to move this project into production. 

Comparing bets between two data streams is handled by the compareGetBets method. This method plays a vital role in maintaining data consistency and integrity across different streams. By comparing the bets from two distinct sources, we can identify discrepancies and ensure that our data remains reliable. This method involves a call that systematically compares the bets, highlighting any differences and enabling us to address them promptly. 

public void compareGetBets(Set<String> betIds, List<Bet> betList) { executorService.submit(() -> { if (useDatastoreToggle.isEnabled()) { List<Bet> betReportingBets = betReportClientDelegate.getBets(betIds) .join();  compareBets(betReportingBets, betList); } else { betDatastoreDelegate.getBets(betIds) .thenAccept(betDatastoreBets -> compareBets(betList, betDatastoreBets)); } }); }

Cashout Success Rate Comparison 

These statistics underscore the tangible impact of our initiatives, highlighting significant improvements in cashout success rates during high-profile events like Cheltenham and the Grand National. These metrics serve as a testament to the effectiveness of our strategic preparations and operational enhancements in enhancing customer satisfaction and optimising performance. 

In 2023, during the Cheltenham event, Betfair (BF) boasted a cashout success rate of 54%, while Paddy Power (PP) led with 68%. Similarly, during the Grand National, BF achieved a 59% success rate, while PP slightly surpassed with 73%. Fast forward to 2024, significant improvements were evident. At Cheltenham, BF saw a remarkable increase to 79%, marking a substantial 25% surge, whereas PP experienced a more moderate rise to 77%, up by 9%. The Grand National also witnessed impressive progress, with BF achieving a 78% success rate, a notable 19% increase, and PP reaching an impressive 91%, indicating an 18% improvement. These statistics underscore the tangible impact of our decision to decouple cashout features, resulting in enhanced customer satisfaction and operational efficiency during peak events. 

Team Contributions and Experiences 

To bring our project to life and prepare for important events with significant customer flow, the collective expertise and dedication of our team members have been invaluable. Below are some testimonials and experiences from colleagues who have been instrumental in the success of this project. 

Rui Santos, Associate Software Engineer: 

"Spring Racing represents a significant challenge for all UK&I teams, due to the considerable increase in traffic on our services before, during and after the races. To ensure the quality of the services, it was imperative to identify possible risks and perform multiple performance tests. These performance tests were supported by specific targets, which needed to be aligned with other teams. To facilitate these processes, regular working group meetings were organised to discuss the tests and the associated risks. During the Cheltenham/Grand National days, in addition to the spectacular surroundings and collaboration in the Blip office, it was essential to implement active monitoring to detect possible failures in our services, particularly regarding performance (response times to our clients) and lag (on our streams). The main objectives were achieved, namely the absence of P1 or P2 incidents in our team's services, and the absence of reported problems. For the next few years, it's important to continue working to make Spring Racing smoother and improve certain aspects, such as automating some performance tests."

Filipe Lemos, Software Engineer:

"I work with Rich Content and Live Data, which makes the Spring Racing season always a very challenging event. But this year was special, a new brand had started using our services — SBG (Sky Betting & Gaming). This meant more load, more eyes, we had to be at our A-game. Additionally, we were handing over our services to a new team, which had never seen or touched this flow. Pressure was high but that is also where we as a company thrive. Prior to the event, we have done weekly load tests to confirm that our services were performant with the load expected for those days. During the event, we had all our focus on the races. We had to make sure that every runner had their silks, tips or other relevant datapoints. Every day a new team member was responsible for the sanities and to keep an eye on the production channels for any possible issue. Thankfully, everything went well, and we are ready for the next big event - Euro 2024."

Narciso Caldas, Software Engineer:

"I'm a software engineer at Blip, working in the Bet Building and Placement flow for the Betfair and Paddy Power brands. This flow is responsible for proving and allowing to place betting opportunities. I was the tech lead responsible for the services of this flow during the Spring Racing season (Cheltenham and Grand National). From the beginning, the main objective for this season was to have the most sooth possible events without any major incidents and with 0% downtime in the team's services, providing the best customer experience possible. To achieve it, we needed to ensure that the following topics were aligned with the working group and completed before the first event: 

• Have realistic targets defined for the services; 
• Service capacity assessed and capable of handling the targets; 
• Risks and action plans defined to act in case of issues; 
• Alignment with the teams of dependent services about the previous topics.

During the Cheltenham and Grand National events, the office has a different atmosphere since people from all the teams go there to monitor the services. There is a lot of communication and exchange of information between the teams to help each other and have fun. During racing days, the monitoring activities consisted of looking at and analysing metrics and logs from the services to act rapidly and minimise the impact on the customers if we found any issues. Thankfully, we had a smooth Spring Racing season without any major issues, which paid off all the effort we put into the preparation and monitoring and was our objective for the festival. Having a clear plan of action on what to do daily and if something goes wrong and good communication with all the working group was essential to achieve a smooth festival and great experience from our services."

Conclusion 

Our strategic initiatives and meticulous preparations have proven instrumental in ensuring peak performance and customer satisfaction during major sports events like Cheltenham and the Grand National. By optimising platform performance and decoupling cashout options, we streamlined our operations, reduced costs, and significantly enhanced our cashout success rates. Through collaborative efforts across our technical teams, we maintained flawless system operations, swiftly addressing challenges in real-time, and ultimately delivering exceptional technical solutions that meet the rigorous demands of our industry. Looking ahead, we remain committed to refining our processes and leveraging our learnings to further elevate the experience for our customers in future events.

We acknowledge the expertise and input of João Marques (Associate Engineering Manager), Daniel Silva (Associate Engineering Manager), João Basto (Software Engineer), and the other colleagues mentioned in the testimonials throughout the development of this article.

Disclaimer: The following article content and excerpts are derived from Manning Publications' article titled "Modern Concurrency with Go”. While we have restructured and integrated portions of their original content, both text and images, into our own discussion, we want to acknowledge the original author's contributions. We have taken care to provide proper attribution and references to the original source. The primary focus of this Blip’s article repost is to highlight the expertise and contributions of James Cutajar, our Principal Software Engineer and Learn Concurrent Programming with Go book author, in our context and to provide insights on the topic that are relevant to our audience. We encourage readers to explore the original article and the published book for a deeper understanding of the subject matter.

Introduction 

In today's tech landscape, efficient and scalable solutions are crucial for modern applications. Concurrent programming offers a pathway to improved performance and scalability. In this article, we'll explore the first chapter of the book Learn Concurrent Programming with Go. Come along as we explore James’ point of view of concurrent execution, scaling programs, and why Go is the language of choice for concurrency.

About the Author and the Book 

James Cutajar is dedicated to backend development at Blip, part of the Flutter group, and has recently published a book that provides a practical, hands-on introduction to creating software for modern multiprocessor systems. In this book, the author guides the reader on how to divide larger programming tasks into independent parts that can run simultaneously, on how to use the Go language to implement common concurrency patterns by utilising readers-writer locks, semaphores, message passing, and memory sharing.

Learn Concurrent Programming with Go explores the principles and real-world applicability of concurrent programming with Go. Unlike sequential programming, concurrent programming leverages multiple CPU cores, boosting program execution speed and overall efficiency. 

“Learn Concurrent Programming with Go” Deep Dive 

Interacting with a concurrent world 

We live and work in a concurrent world. The software that we write models complex business processes that interact together concurrently. Even the simplest of businesses typically have many of these concurrent interactions. For example, we can think of multiple people ordering online at the same time or a consolidation process grouping packages together while simultaneously coordinating with ongoing shipments.

Concurrent programming is about writing instructions so that multiple tasks and processes can execute and interact at the same time.  

Increasing throughput 

For the modern developer it is ever more important to understand how to program concurrently. This is because the hardware landscape has changed over the years to benefit this type of programming. 
Prior to multicore technology, processors’ performance increased proportionally to clock frequency and transistor count, roughly doubling every 2 years. Processor engineers started hitting physical limits due to overheating and power consumption, which coincided with the explosion of more mobile hardware such as notebooks and smartphones. To reduce excessive battery consumption and CPU overheating while increasing processing power, the engineers introduced multicore processors. 
In addition, with the rise of cloud computing services, developers have easy access to large, cheap processing resources to run their code. All this extra computational power can only be harnessed effectively if our code is written in a manner that takes full advantage of the extra processing units. 

Having multiple processing resources means we can scale horizontally. We can use the extra processors to compute executions in parallel and finish our tasks quicker. This is only possible if we write code in a way that takes full advantage of the extra processing resources. 
What about a system that has only one processor? Is there any advantage in writing concurrent code when our system does not have multiple processors? It turns out that writing concurrent programs is a benefit even in this scenario. 
Most programs spend only a small proportion of their time executing computations on the processor. Think for example about a word processor that waits for input from the keyboard. Or a text files search utility spending most of its running time waiting for portions of the text files to load in memory. We can have our program perform a different task while it’s waiting for input/output. For example, the word processor can perform a spell check on the document while the user is thinking about what to type next. We can have the file search utility looking for a match with the file that we have already loaded in memory while we are waiting to finish reading the next file into another portion of memory. 
Think for example when we’re cooking or baking our favorite dish. We can make more effective use of our time if, while the dish is in the oven or stove, we perform some other actions instead of idling and just waiting around. In this way we are making more effective use of our time and we are more productive. This is analogous to our system executing other instructions on the CPU while concurrently the same program is waiting for a network message, user input, or a file writing to complete. This means that our program can get more work done in the same amount of time.

Improving responsiveness 

Concurrent programming makes our software more responsive because we don’t need to wait for one task to finish before responding to a user’s input. Even if we have one processor, we can always pause the execution of a set of instructions, respond to the user’s input, and then continue with the execution while we’re waiting for the next user’s input. 
If again we think of a word processor, multiple tasks might be running in the background while we are typing. There is a task that listens to keyboard events and displays each character on the screen. We might have another task that is checking our spelling and grammar in the background. Another task might be running to give us stats on our document (word count, pages, and so on). All these tasks running together give the impression that they are somehow running simultaneously. What’s happening is that these various tasks are being fast-switched by the operating system on CPUs. 

Programming concurrency in Go 

Go is an ideal language to use to learn about concurrent programming because its creators designed it with high-performance concurrency in mind. The aim was to produce a language that was efficient at runtime, readable, and easy to use. This means that Go has many tools for concurrent programming. Let’s take a look at some of the advantages to using Go for concurrent programs. 

Goroutines at a glance 

Go uses a lightweight construct, called a goroutine, to model the basic unit of concurrent execution. As we shall see in the next chapter, goroutines give us a hybrid system between operating system and user-level threads, giving us some of the advantages of both systems. 
Given the lightweight nature of goroutines, the premise of the language is that we should focus mainly on writing correct concurrent programs, letting Go’s runtime and hardware mechanics deal with parallelism. The principle is that if you need something to be done concurrently, create a goroutine to do it. If you need many things done concurrently, create as many goroutines as you need, without worrying about resource allocation. Then depending on the hardware and environment that your program is running on, your solution will scale. 
In addition to goroutines, Go gives provides us with many abstractions that allow us to coordinate the concurrent executions together on a common task. One of these abstractions is known as a channel. Channels allow two or more goroutines to pass messages to each other. This enables the exchange of information and synchronization of the multiple executions in an easy and intuitive manner. 

Modelling concurrency with CSP and primitives 

The other advantage of using Go for concurrent programming is its support for Communicating Sequential Processes (CSP). This is a manner of modelling concurrent programs in order to reduce the risk of certain types of programming errors. CSP is more akin to how concurrency happens in everyday life. This is when we have isolated executions (processes, threads, or goroutines) working concurrently, communicating to each other by sending messages back and forth. 
The Go language includes support for CSP natively. This has made the technique very popular. CSP makes our concurrent programming easier and reduces certain types of errors. 

Sometimes the classic concurrency primitives found in many other languages used with memory sharing will do a much better job and result in a better performance than using CSP. These primitives include tools such as mutexes and conditional variables. Luckily for us, Go provides us with these tools as well. When CSP is not the appropriate model to use, we can use the other classic primitives also provided in the language. 
It’s best to start with memory sharing and synchronization. The idea is that by the time you get to CSP, you will have a solid foundation in the traditional locking and synchronization primitives. 

Scaling performance 

Performance scalability is the measure of how well our program speeds up in proportion to the increase in the number of resources available to the program. To understand this, let’s try to make use of a simple analogy. 
Imagine a world where we are property developers. Our current active project is to build a small multi story residential house. We give your architectural plan to a builder, and she sets off to finish the small house. The works are all completed in a period of 8 months. 
As soon as we finish, we get another request for the same exact build but in another location. To speed things up, we hire two builders instead of one. This second time around, the builders complete the house in just 4 months. 
The next time that we get another project to build the same exact house, we agree to hire even more help so that the house is finished quicker. This time we pay 4 builders, and it takes them 2 and a half months to complete. The house has cost us a bit more to build than the previous one. Paying 4 builders for 2.5 months costs you more than paying 2 builders for 4 months (assuming they all charge the same). 
Again, we repeat the experiment twice more, once with 8 builders and another time with 16. With both 8 and 16 builders, the house took 2 months to complete. It seems that no matter how many hands we put on the job, the build cannot be completed faster than 2 months. In geek speak, we say that we have hit our scalability limit. 
 

Amdahl’s Law 

Amdahl’s Law states that the overall performance improvement gained by optimizing a single part of a system is limited by the fraction of time that the improved part is actually used. 

Amdahl’s law tells us that the non-parallel parts of an execution act as a bottleneck and limit the advantage of parallelizing the execution. The image below shows this relationship between the theoretical speedup obtained as we increase the number of processors. 

Gustafson’s Law 

In 1988 two computer scientists, John L. Gustafson and Edwin H. Barsis, reevaluated Amdahl’s law and published an article addressing some of its shortcomings. It gives an alternative perspective on the limits of parallelism. Their main argument is that, in practice, the size of the problem changes when we have access to more resources. 
If we were developing some software and we had a large number of computing resources, if we noticed that utilizing half the resources resulted in the same performance of that software, we could allocate those extra resources to do other things, such as increasing the accuracy or quality of that software in other areas. 
The second point against Amdahl’s law is that when you increase the problem size, the non-parallel part of the problem typically does not grow in proportion with problem size. In fact, Gustafson argues that for many problems this remains constant. Thus, when you take these two points into account, the speedup can scale linearly with the available parallel resources. 

Gustafson’s Law tells us that as long as we find ways to keep our extra resources busy, the speedup should continue to increase and not be limited by the serial part of the problem. This is only if the serial part stays constant as we increase the problem size, which according to Gustafson, is the case in many types of programs. 
 

Summary 

• Concurrent programming allows us to build more responsive software. 
• Concurrent programs can also provide increased speedup when running on multiple processors. 
• We can also increase throughput even when we have one processor if our concurrent programming makes effective use of the input/output wait times. 
• Go provides us with goroutines which are lightweight constructs to model concurrent executions. 
• Go provides us with abstractions, such as channels, that enable concurrent executions to communicate and synchronize. 
• Go allows us the choice of building our concurrent application either using concurrent sequential processes (CSP) model or alternatively using the classical primitives. 
• Using a CSP model, we reduce the chance of certain types of concurrent errors; however, certain problems can run more efficiently if we use the classical primitives. 
• Amdahl’s Law tells us that the performance scalability of a fixed-size problem is limited by the non-parallel parts of an execution. 
• Gustafson’s Law tells us that if we keep on finding ways to keep our extra resources busy, the speedup should continue to increase and not be limited by the serial part of the problem. 

Learn more about Concurrent Programming with Go here.

In this article, we invite you to explore one of the standout presentations of the Product Weekend event that emerged from one of our Blippers - Margarida Brandão, Lead Product Manager, about “Managing Core Internal Products”.

Product Weekend Hosted at Blip 

Our company recently played host to a transformative Product Weekend and became the epicentre of Product knowledge, welcoming presenters from various tech powerhouses to contribute with their unique perspectives and skill sets. It was the second edition of the event in Porto and we had the honour of being part of this event that aims to help young Product Managers learn from experienced experts and peers, build strong relationships and boost their careers. The event showcased our commitment to fostering cross-industry collaboration but also highlighted our dedication to providing a conducive platform for meaningful discussions and synergy within the community.

Deep Dive on Managing Core Internal Products

Margarida's presentation delved into the exciting realm of managing trading products within the gaming and betting industry, with insights that illuminated the intersection of technology and entertainment. The atmosphere was focused on the importance of supporting internal products, that are the base of business, and on having a product-focused mindset, regardless of the area.

Defining Trading

Trading, within the context of our operations, goes beyond the mere pursuit of aspirational business outcomes. It is a multifaceted approach that places paramount importance on both supporting ambitious business results and prioritising the ultimate end-user experience. In this intricate framework, our focus is distinctly directed towards enhancing the experience for our internal customers, namely the Traders. By aligning our efforts with the needs and aspirations of these key stakeholders, we aim to create an environment where optimal trading activities thrive, fostering a symbiotic relationship between our business goals and the satisfaction of our internal customers.

The Road to Global Trading Team

Integration of PPB and SBG Brands for Global Trading:

In a strategic move, the Risk and Trading products, integral to the operations of all global brands, were predominantly housed within our Paddy Power Betfair (PPB) and Sky Betting and Gaming (SBG) brands. Recognising the synergies and collective strengths of these entities, the Global Trading Team was conceived. This innovative approach involved uniting the expertise and offerings of the PPB and SBG teams, harnessing the comprehensive suite of products they individually possessed. By consolidating these resources and fostering collaboration, the establishment of the Global Trading Team aimed to create a unified and robust platform that could effectively serve the diverse needs of all global brands. 

Embracing Innovation:

In the quest to distinguish themselves and carve a unique identity, the recently established team opted for an unconventional and innovative operational methodology. Faced with the complexities of managing multiple brands and an increased scope of responsibilities, the challenges they encountered were more formidable than ever before. 

New Product Lifecycle Approach:

This forward-thinking team challenged the conventional approach to product management. Instead of adhering to the traditional project-based model dictated by senior management priorities, they shifted their focus towards a holistic perspective on the complete product lifecycle. This distinctive approach involved taking ownership of the entire product journey, from inception to delivery, marking a departure from the conventional project-centric mindset. By concentrating on the full product lifecycle, the team aimed to foster a culture of continuous improvement and adaptability, aligning their strategies with a more comprehensive and forward-looking product management philosophy.

Guiding Principles

Autonomy and Ownership:

Product-focused teams are granted autonomy to take ownership of their responsibilities, understanding their objectives and having the authority to determine the subsequent steps. Emphasising cycles of building, measuring, and learning, these teams cultivate trust with leadership by prioritising transparency in their actions.

Three Perspectives for Optimal Collaboration:

The synergy of product, engineering, and delivery perspectives is the sweet spot for effective teamwork. Ensuring continuous representation from these three facets maximises the overall value delivered by fostering a comprehensive and collaborative approach to decision-making.

VMOST and OKRs for Effective Communication:

Employing VMOST (Vision, Mission, Objectives, Strategies, and Tactics) and OKRs (Objectives and Key Results) serves as a robust communication strategy. Establishing a single source of truth for the team's strategy and value streams, these frameworks provide clarity and coherence, helping make sense of both small and significant tasks through the alignment with cascading objectives. 

Lessons Learned 

Developing a genuine product mindset is a gradual process that demands dedication and perseverance. However, the rewards become apparent as interactions with stakeholders and leadership become more engaging and fruitful. In situations where dependencies between teams are unavoidable, fostering alignment and collaboration can be achieved by sharing OKRs across teams. This collaborative approach helps in promoting a unified goal and streamlining efforts. Moreover, prioritising the optimisation of product value necessitates addressing internal issues first, akin to cleaning the house before expecting optimal outcomes. This foundational step ensures a solid base for product-focused initiatives and enhances the overall effectiveness of the product development process. 

Challenges 

These challenges come up in managing Trading products or any kind of internal product. 

• Avoiding Disruption to Critical Processes: We oversee products that constitute the core of our business and serve as the bedrock for Flutter, rather than being merely attractive features for end users that may not succeed. Consequently, we cannot afford to introduce issues, making it challenging for product teams to experiment and innovate. 
• Data Visibility Troubles: This area has suffered from insufficient investment due to the internal nature of our products. Our struggle lies in understanding how our products are utilised and their performance due to a lack of data. We are actively working to rectify this situation. 
• UX in the Backseat: Another neglected area is the lack of investment in User Experience and User Interface design, primarily because our products are internal tools essential for users to perform their tasks. The consequence is disjointed user journeys that impact productivity. Our current focus is on addressing and improving this aspect. 
• A/B Testing Limitations: Traditional product management tools like A/B testing are impractical for us, demanding thousands of users when we only have a few hundred. This limitation hampers our ability to experiment and conduct statistical analysis, leading us to heavily rely on qualitative feedback, which is not always sufficient. 
• Unseen Impacts in a Complex Chain: Being part of an intricate chain of products and systems, it is challenging to discern the impact of the changes we implement due to the sheer volume of activities. Additionally, our influence on revenue is not always direct, making it difficult to precisely estimate our impact. 
• Resisting Change When It Works: Investment in areas such as redesigns, scalability, and stability often take a backseat, as the challenge lies in prioritising these initiatives. The mindset of "if it's working, why change it?" prevails, adding another layer of complexity to our investment decisions. 

Opportunities That Come From Challenges 

Navigating challenges presents unique opportunities within this intriguing domain, surpassing the complexity of conventional e-commerce. The product operates on an extraordinary scale, bringing about substantial impact and, consequently, introducing its own set of challenges that contribute to the excitement of the job. Engaging directly with users offers the chance to delve into their perspectives, inquire about their experiences, validate ideas, and gain profound insights into the product's usage. This direct interaction serves as a valuable source of learning and adaptation. Furthermore, the dynamic landscape provides ample opportunities to drive change and foster innovation, allowing individuals to play a pivotal role in shaping the future trajectory of the product. 

Key Insights 

Each product domain is unique, and there is no one-size-fits-all framework: The approach to product management must be tailored to fit the specific nuances of our business area, customers, and organisational structure. Numerous frameworks and tools exist, and experimentation is acceptable; it's crucial to try different approaches and discard them if they do not align with the intended goals. 

Making decisions based on data is crucial for success: While progress can be achieved without data, incorporating it into the decision-making process enhances the potential for more significant accomplishments and continuous learning. Therefore, data should be a central element in effective product management. 

Understanding your customer is paramount: Identifying and prioritising primary customers amid the myriad voices within a complex organisation is essential. Despite the various perspectives, customers inherently comprehend their needs, making it imperative to prioritise their insights and preferences. 

Product Managers possess a unique opportunity to drive change: Their ability to grasp the broader picture, think strategically and resourcefully, and establish meaningful relationships positions them as catalysts for progress within an organisation.

Blip has a dedicated Security Testing team that performs penetration testing, continuous testing, and red teaming. By being part of S-SDLC (Secure Software Development Life Cycle) process, the team handles development and infrastructure security vulnerabilities. During the development phase, penetration tests are carried out to detect and correct vulnerabilities and ensure that new products are launched into production with a high-level of security.

Introduction

In the dynamic realm of cybersecurity, where the battle between defenders and adversaries intensifies daily, the discovery of vulnerabilities is both a triumph and a challenge.

In this blogpost, we rev our engines and delve into the world of race conditions in security testing, where the unexpected often lies hidden beneath layers of code. Our spotlight is on a challenge encountered during a penetration test – the discovery of a race condition. We'll explore the practical aspects of identifying a race condition, and the key takeaways it brings to the forefront in our ongoing battle against threats, emphasising the importance of proactive testing in maintaining a robust digital infrastructure. 

What Is a Race Condition? 

Developers and testers often encounter the challenge of race conditions. These vulnerabilities, similar to a high-speed race in the digital domain, can introduce security loopholes that may compromise the integrity of systems.  

In the context of penetration testing, a race condition is a security flaw that arises when a system's behaviour is affected by the order or timing of particular operations taken during the test. A race condition occurs when two or more threads can access shared data and they try to change it at the same time. Think about two threads running the same program and working with the same variables - the order in which the operations happen will lead to unexpected behaviours. 

An attacker may use this kind of vulnerability to change the way things are supposed to happen, which could result in data breaches, illegal access, or other security lapses. To evaluate a system's resistance to simulated attacks, penetration testers frequently look for and take advantage of race circumstances. By doing this, they assist organisations in identifying and fixing vulnerabilities before hostile actors may use them in actual attacks.  

Application Logic 

In an application that stores an integer X value, a new Y value can be set, but there are a few things to consider: 

Assuming there is a default starting X value: 

• By choosing to set a higher Y value, then the value will be kept on hold during a period before being set. 
• By choosing to set a lower Y value, the value is immediately set. 
• The application continuously tracks the ACTIVE value and the PENDING value (the latter can be null). 

After observing this behaviour, a simplified version of the code would be close to this:

def function(int y): 
  if (y > x): 
    wait_set(y) 
  else: 
    set(y)

Two things can happen here depending on the conditional statement: 

If Y is larger than X –> ACTIVE = x and PENDING = y 
If Y is not larger than X –> ACTIVE = y and PENDING = null (this also applies if the variables are equal, but that won’t matter too much)

The Problem 

Suppose we launch two threads of the program at the same time and the variables are shared. So, if thread 1 sets a value for Y, it is also set for thread 2. That being said, the odds of something weird happening seem null, since it requires super precise timing, right? Not quite. We will go over how James Kettle made this so much easier to exploit later. But first, let’s go back to our problem - what if we could fail the if statement (Y value lower than X) and then immediately change Y to a value higher than X mid execution, so when the line set(y) is ran, a value higher than X will become active without a cooldown? 

Looking at the diagram below, on the left we have thread 1, on the right thread 2. The initial values are shown above the code, which is there for quick reference. 

⚪ Starting with ACTIVE = 50 and PENDING = null 

🔴 T1 sets y = 49 

🔴 T1 fails the if condition since 49 > 50 ? False 

🟢 T2 sets y = 51 

🔴 T1 calls set(y), which translates to set(51) since T2 just set the Y variable to 51 –> ACTIVE = 51 and PENDING = null 

🟢 T2 passes the if condition since 51 > 50 ? True 

🟢 T2 calls wait_set(51) –>  ACTIVE = 51 (same as the latest value) and PENDING = 51

Burp Suite’s repeater tab was used to send these requests by putting both tabs in a folder and selecting Send group in parallel (single-packet attack). The example below is sending the three requests in the limit-overrun folder using this technique.

It took several attempts of sending two requests in a single packet (one with a higher value than ACTIVE, other with a lower value) to make this actually work. The server would act as if only one of the values was sent. No pattern or consistent behaviour was noted, but one of the values would indeed be set. It would either overwrite ACTIVE (lower value) or set a new PENDING (higher value), depending on which one the server “chooses”. To not reuse the same values, after every attempt the higher value was incremented and the lower value decremented - for example, send 51 and 49, then 52 and 48, 53 and 47 and so on. 

After many attempts, the response size suddenly shot up. By taking a more careful look, two ACTIVE variables were observed. One set to the higher value and another one set to the lower one. Even if a request that fetched the current values was sent, the response would be something like this: 

ACTIVE = 51 
ACTIVE = 49 
PENDING = null

Understanding What Happened 

After getting 2 ACTIVE variables, setting up a new value was attempted, but the server response was a 500 Internal Server Error. The same happened when performing operations that required these values. Even after resetting the environment and repeating the attack, the behaviour was consistent. 

Timing Is The Key 

A race condition is difficult to exploit because there are a lot of timing variables involved, and one has to be lucky enough to have each single line executed in a specific order. Taken from James Kettle: 

Even if the requests are coded to be sent at the exact same time, there’s the network latency, the jitter (time between transmission and reception of data) and the server’s latency. 

After some iterations, James’ final solution is to send several requests in a single TCP packet. Which means the jitter and latencies involved will all be the same. There’s still some trial and error due to the processing times on the server, but the effectiveness of this attack goes up exponentially.  

Other techniques were attempted, such as using Burp Suite’s intruder and last-byte sync, but the only time interesting results were obtained was when the requests were sent in a single TCP packet. 

Conclusion 

Applications that involve concurrent processing can easily incur this type of vulnerability. One way to avoid them is to design the application with the concept of transactions, in which parallel transactions that share resources can include several operations each, but one transaction is only executed when no other transaction is taking place (atomic execution). Another way to reduce the risk of this vulnerability and other types of attacks is to implement a "rate limit" in which the user must wait a few seconds before sending a second operation. 

Our experience with penetration testing emphasises the importance of diligent security checks. By fixing flaws like race conditions, we prepare our systems to handle both simulated and real-world threats effectively. That being said, encountering situations like this is never a setback but always chance to improve.

Are you a Security Testing Engineer looking to join our team? Check out the opportunity we have available.

gRPC is a modern, open-source Remote Procedure Call (RPC) framework that can run in any environment. It enables client and server applications to communicate transparently, and makes it easier to build connected systems. This blog post explores the key features and benefits of gRPC, compares it with REST, and explains how you can use it in your projects.

Before gRPC, Let’s Explain RPC

RPC extends conventional local procedure calling so that the called procedure doesn’t need to live in the same host/address space as the calling procedure. The two communicating processes might be on the same system or they can live in different systems with a network connection in the between.

In other words it’s like a form of client-server communication that uses a function call rather than an usual HTTP call. It makes use of IDL (interface definition language) as a contract on functions called and data types returned.

gRPC mirrors this architectural style of client-server communication also using function calls. So, in reality, gRPC isn’t the only fish in the sea, but it adopted this technique and made it better in a way that made it super popular now.

What is gRPC?

gRPC doesn’t stand for Google Remote Procedure Call as many people might think. So what the g actually stands for?

Google changes its meaning in every version they release. g actually started meaning gRPC, and then it evolved to good, green, gentle, and more – to the point that they even wrote a README file to list all the meanings.

gRPC is a high-performance, open source RPC framework released by Google in 2015. It’s currently a Cloud Native Computing Foundation project and Google has been using a lot of the underlying technologies and concepts for a long time. Several of Google’s cloud products use the current implementation.

Companies such as Square, Netflix, CoreOS, Docker, CockroachDB, Cisco, Juniper Networks have been also using this technology.

It enables communication between client and server applications using a simple and efficient protocol, making it ideal for building distributed systems and microservices.

gRPC uses Protocol Buffers (aka protobuf) as its default serialisation framework and HTTP/2 as its underlying transport protocol, providing high performance and efficiency.

The gRPC architecture entails a client-server model. A client sends the server a request and the server replies back.

The communication between both is responsibility of gRPC application and its corresponding service definitions. These service definitions in .proto files contain information on the methods, arguments, and return types.

The application and its code is auto-generated and acts as a client-side proxy making remote calls look and behave like local function calls.

The gRPC framework abstracts away the complexity of the communication. You know nothing more than the client calls the service method, and the method runs on the server. The magic that plays out in-between isn’t open to you.

So why is this so popular? Let’s take a look at some key benefits and features.

Features and Benefits

High Performance Transfers

gRPC makes use of a binary serialisation format (Protocol Buffers) which results in faster serialization and parsing compared to traditional text-based formats like json or xml as well as smaller message sizes. This makes the transmission more efficient, with lower resources while still saving bandwidth.

json payload (79 bytes):

{
  "age": 35,
  "first_name": "Renato",
  "last_name": "Cardoso"
}

protobuf payload (19 bytes):

message Person {
  int32 age = 1;
  string first_name = 2;
  string last_name = 3;
}

Multiplexing

gRPC makes use of HTTP/2 protocol, which is a big enhancement over HTTP/1.1. HTTP/2 introduces features such as request and response multiplexing over a single connection, header compression and server push. This results in a reduction on latency and network transfer speeds while improving overall performance.

Language-Agnostic & Code Generation

gRPC uses Protocol Buffers to define messages and services, which compile into code in a variety of programming languages including Java, C++ and Python. This makes communication between services simple and flexible regardless of the language used for development. It also allows developers to compile their code in their preferred language while reducing the amount of boilerplate code needed to write.

Cross-Platform

gRPC supports running on multiple platforms, including Linux, Windows, and macOS. This makes it easier to develop and deploy services on a variety of environments.

Secure

gRPC considers security to be a first class citizen. It supports built-in SSL/TLS encryption for secure transfers and mutual authentication.

Streaming Support

gRPC provides a number of different types of APIs, including unary, server streaming, client streaming and bidirectional streaming. These APIs allow for a variety of different communication patterns between the client and server.

• Unary RPC: in this case, the client sends a request message to the server and receives a response - simplest form of RPC.
• Server streaming RPC: in this case, the client sends a request message to the server and receives a sequence of responses.
• Client streaming RPC: in this case, the client sends a sequence of messages and receives a single response from the server.
• Bidirectional streaming RPC: in this case, the client and the server exchange messages in both directions. This one is the most complex because client and server keep sending and receiving multiple messages in parallel and with arbitrary order. It’s flexible and non-blocking which means both sides don’t need to wait for the response before sending the next messages.

Strong Community Support

Because it’s developed by Google and gained a lot of traction by the community, gRPC is being widely adopted and keeps being continuously improved by developers and organisations.

When To Use gRPC Over REST

gRPC and REST are great for building APIs, and the decision of when to use gRPC over REST really depends on your requirements.

Here are some use cases in which gRPC might be a better choice:

1. Performance and efficiency: If you need low-latency communication with high throughput, gRPC is the way to go. It’s designed for it since it uses HTTP/2 with support for multiplexing and binary serialization making it more efficient in terms of data transfer and latency.
2. Strongly typed contracts: gRPC relies on protocol buffers for service contracts definition. This allows for strong typed APIs with clear data structures and services, enabling better code generation, type safety and easier evolution.
3. Code generation: If you want to be fast and decouple the programming language from the protocol, gRPC has your back. It supports automatic code generation for client and server which can save development time and reduce the chances of making errors.
4. Bidirectional streaming: gRPC supports bidirectional streaming where both the client and server can send multiple messages in parallel. This is useful for use cases like live data streaming, chats or even notifications.
5. Middleware and interceptors: gRPC provides great support for implementing middleware and interceptors allowing you to add functionality like authentication, logging and monitoring in a consistent way.

Nevertheless, REST might be a better choice in other situations:

1. Simplicity and wider adoption: REST is simpler to set up and can be an excellent choice for small/internal APIs or simple services. It’s currently widely adopted and understood in the industry.
2. Compatibility: REST is well suited for integration with existing systems, as it’s supported by nearly all programming languages and platforms being one of the best examples the support of all current browsers.
3. Readability: REST APIs use human-readable formats like json or xml, which can be helpful for debugging and manual testing.

Ultimately, the choice between both depends on your project’s requirements. You could as well use a combination of both as long as it makes sense for your particular scenario. If you prioritize performance, efficiency, and strong typing, gRPC may be the better choice. If simplicity and compatibility with existing systems are more critical, REST may be the way to go.

Why We Use gRPC at Flutter

Flutter is a global sports betting, gaming, and entertainment provider. We operate some of the world’s most innovative, diverse, and distinctive brands with over 18 million customers worldwide.

Our customers place billions of bets every year across all brands. Each brand operates individually however Flutter needed to have a centralised place where it could collect every placed bet and process it centrally.

To achieve it, we decided to build an application called GBSC (Global Bet Stream Collector) which has the following architecture. For simplicity let’s look at some brands only.

The requirements were very clear and well defined. Let’s look at the most important ones:

1. Ability to ingest a big amount of bets that may come from various brands operating in distinct parts of the world (countries/continents).
2. Data transmission must have as low latency as possible because there might exist some automated actions right after processing the data.
3. Individual access control for each brand.
4. Same Bet model used across all brands in order to deal with data in a single format.
5. Possibility of model evolution.
6. Give the possibility for each brand to choose their preference in what concerns the place to deploy their integration application, the language of choice and as well as the operating system.
7. Data encryption in transit.

Looking at these requirements it’s quite easy to understand gRPC ticks pretty much all of them. So relating to the previous requirements gRPC answers to them:

1. By using unary or bidirectional streaming calls, it can achieve this with the help of persistent connections in addition to asynchronous API calls.
2. By using HTTP/2 persistent connections, it’s able to perform SSL handshake just once avoiding all the network round trips that normally exist in this process. Using a binary data model also allows to keep message size smaller.
3. By using interceptors, it can easily validate against some HTTP headers (e.g. Authorisation header with JWT token) in order to authorise the call.
4. By using protobuf, it can easily build a library in any gRPC supported language and distribute it to the brands, in a way that they can import it on their projects. It’s definitely better approach than distributing the protobuf files themselves.
5. By using protobuf, it can use its backwards and forwards compatibility. As long as we follow some simple practices when updating .protodefinitions, old code reads new messages without issues, ignoring any newly added fields. To the old code, deleted fields have their default value, and deleted repeated fields are empty. We currently make use of proto-backwards-compat-maven-plugin to help us not make mistakes.
6. By using protobuf, it can generate code in any language supported by gRPC. This provides great flexibility for each brand to opt by its language of choice and its deployment preference. gRPC is also supported in all big cloud providers.
7. The HTTP/2 protocol and TLS/SSL encrypts data in-transit which helps to mitigate spoofing attacks by robustly encrypting and authenticating transmitted data, preventing interception of traffic and blocking the decryption of sensitive data on the bet payload or authentication tokens.

Using gRPC

In order to use gRPC in your own project, you need to define your services and messages using a .proto file. This file defines the methods and parameters for your services, as well as the messages exchanged between the client and server.

Once you’ve defined your .proto file, you can use the gRPC tools to generate client and server code in your preferred programming language.

To run your gRPC service, you need to start a gRPC server that listens for incoming requests. Then, you can then start your client and make requests to the server.

Let’s look at a service definition in detail:

bet-service.proto

syntax = "proto3";

package com.flutter.gbs;

import "bet.proto";

service BetService {
  
  // Unary call to send a Bet message and receive a BetAck message back
  rpc sendBet(Bet) returns (BetAck);
  
  // Bi-directional stream calls to send multiple Bet messages and receive a stream of BetAck messages back
  rpc sendBets(stream Bet) returns (stream BetAck);
}

This code block shows a RPC service BetService definition that inside declares 2 methods: sendBet and sendBets (notice the service keyword before declaring each service and rpc keyword before any method).

Each method has an input argument as well as a response. Besides the name, the big difference between the two is the keyword stream. This tells you that on the sendBets method you are making use of bidirectional streaming of request and response messages instead of the traditional unary calls.

bet.proto

syntax = "proto3";

package com.flutter.gbs;

message Bet {

    // Unique ID from brand bet platform used to identify the bet placed
    string betId = 1;
    
    // Timestamp of when the bet was placed
    int64 betTime = 2;
    
    // Amount staked on the bet
    double totalStake = 3;
}

message BetAck {

    // Status of the response, enum of OK or ERROR
    Status status = 1;

    // Error message if occurred
    string error = 2;

    enum Status {
        OK = 0;
        ERROR = 1;
    }
}

Here is the definition of the Bet and BetAck message types. Each property has its type, which can be primitive, or another custom type, such as Status.

After compiling this, the server implementation could be as follows. Here you are just building a successful response without any business logic - this is for demonstration purposes.

BetService.java

package com.flutter.gbs;

import com.flutter.gbs.BetOuterClass.Bet;
import com.flutter.gbs.BetOuterClass.BetAck;
import com.flutter.gbs.BetServiceGrpc;
import io.grpc.stub.StreamObserver;
import lombok.extern.slf4j.Slf4j;

@Slf4j
public class BetService extends BetServiceGrpc.BetServiceImplBase {

    public BetService() {
    }

    @Override
    public void sendBet(Bet bet, StreamObserver<BetAck> responseObserver) {
        responseObserver.onNext(
                BetAck.newBuilder()
                        .setStatus(BetAck.Status.OK)
                        .build());
    }

    @Override
    public StreamObserver<Bet> sendBets(StreamObserver<BetAck> responseObserver) {
        return new StreamObserver<Bet>() {
            @Override
            public void onNext(Bet bet) {
                responseObserver.onNext(
                        BetAck.newBuilder()
                                .setStatus(BetAck.Status.OK)
                                .build());
            }

            @Override
            public void onError(Throwable t) {
                log.error("Received error from streaming client", t);
                responseObserver.onError(t);
            }

            @Override
            public void onCompleted() {
                log.info("Received shutdown from client");
                responseObserver.onCompleted();
            }
        };
    }
}

Conclusion

gRPC and protobuf are powerful tools for building efficient and scalable distributed systems. They provide a fast and efficient way to communicate between services, support backward and forward compatibility, and provide a simple and language-agnostic way to define your data structures. If you are building a distributed system, it’s worth considering using gRPC and protobuf to help you achieve your goals.

Regarding using gRPC or REST, both are useful for building distributed systems, but they differ in their approach to communication and data transfer. REST is simpler and more widely supported, making it a good choice for public APIs or web applications. gRPC is more efficient and supports bidirectional streaming, making it a good choice for internal APIs or high-performance applications.

When choosing between gRPC and REST you should take into account the requirements of your project and choose the technology that best meets your needs. 

Anti-patterns can present serious difficulties in Site Reliability Engineering (SRE), where the objective is to guarantee the stability and dependability of systems. Whether ingrained in procedures, design, or culture, these anti-patterns can obstruct development, complicate reliability initiatives, and even result in system breakdowns. However, promoting sustainable operations and upholding a robust infrastructure requires an understanding of these obstacles and an effective strategy to overcome them.

Celebrating Success: Our Journey at SRE Day London 2023  

It's not only a desire, but a requirement to keep ahead of the curve in the ever-changing world of technology. Our organisation was pleased to take part in the SRE Day in September, which took place in London. The most brilliant minds in the field came together for this international event, which served as a forum for idea sharing, experience sharing, and celebration of the milestones that will influence technology's future. 

Our Principal SRE Engineer Ricardo Castro took center stage as a marker to our commitment to excellence. He gave a compelling talk that emphasised our effort to furthering the area of Site Reliability Engineering. This post will guide you through Ricardo's talk and provide insights into the impact of our team's commitment to this significant event and the priceless knowledge shared. Come celebrate with us as we explore the core of innovation and the progress we're making toward building a dependable and resilient IT ecosystem. This talk describes the anti-pattern of rebranding traditional operations as "SRE."

SRE Origin  

Early in the new millennium, Google introduced SRE, which developed in response to the difficulties brought on by the company's explosive expansion and the growing complexity IT operations. One set of engineers, Ben Treynor included, came up with the term "Site Reliability Engineering" to characterise their creative solution to the problem of maintaining high levels of performance and reliability for services like Google Search. The SRE approach, which took its cues from software engineering, aimed to combine the operational rigor of traditional IT jobs with software development skills. Through the integration of coding, automation, and continuous improvement into infrastructure management, the Google SRE team showcased the efficacy of this proactive, cooperative approach in attaining exceptional system reliability on a large scale. Because of SRE's success at Google, the tech industry has adopted it widely throughout the years, attesting to its effectiveness in preserving high-performing, robust systems in the face of rapidly advancing technological complexity. 

SRE Today 

SRE is at the forefront of assuring the stability and performance of complex systems in today's rapidly changing technological landscape. Organisations from a variety of industries are using SRE as a strategy to overcome the difficulties presented by contemporary distributed architectures, rather than merely as a collection of procedures. By emphasising automation, collaboration, and monitoring, SRE enables teams to improve and proactively manage the dependability of digital services. With companies depending more and more on microservices and cloud-based infrastructures, SRE concepts are essential for improving system performance, cutting down on downtime, and improving user experience in general. 

Today, SRE is not merely a trend but a foundational element that enables companies to deliver resilient and scalable software solutions in the face of dynamic technological landscapes, demonstrating its enduring relevance and importance in the realm of contemporary IT operations. 

Why SRE? 

It is imperative to implement SRE techniques in the rapidly evolving technology landscape of today. By serving as a liaison between development and operations, SRE promotes a culture in software systems that places a premium on efficiency, scalability, and dependability. Businesses may minimise downtime and guarantee a flawless customer experience by adopting SRE principles, which enable them to proactively address and prevent future issues. Beyond standard IT responsibilities, SRE promotes continual improvement, automation, and monitoring to build strong, durable systems.

SRE is essentially a mindset that unites development and operations teams in pursuit of a single objective: providing dependable, high-quality services. It is not merely a methodology. As technology evolves, the need for SRE becomes increasingly apparent, empowering organisations to navigate the complexities of modern infrastructure and meet the growing expectations of users in a rapidly advancing digital era. 

SRE Anti-Patterns: What and Why 

SRE anti-patterns are traps or less-than-ideal SRE practices that might compromise system stability and dependability objectives. These patterns frequently appear when teams misunderstand or improperly implement SRE concepts, which can have unforeseen repercussions like higher downtime, worse performance, or wasteful resource usage.

SRE anti-patterns can include overemphasising certain indicators at the expense of overall system health, overlooking crucial monitoring components, or failing to properly prioritise error budgets. SRE teams must identify and deal with these anti-patterns to maintain alignment with the discipline's fundamental principles and promote an adaptive and continuous improvement culture. Teams can improve their capacity to build and manage robust systems that satisfy user expectations while reducing interruptions and downtime by avoiding SRE anti-patterns. 

How We Are Doing It: Avoiding and Overcoming Anti-Patterns 

The FanDuel brand, part of Blip and the Flutter Group, is a big organisation with a complex environment, so it’s very important how we are approaching these changes and how we are tackling the challenges to avoid anti-patterns.  

As we work to become Site Reliability Engineers (SREs), we understand how critical it is to embrace the concepts of efficiency, scalability, and reliability while also identifying and resolving any potential hazards. We are devoted to finding and addressing SRE anti-patterns in our operations as part of this revolutionary journey. Through vigilant system monitoring and performance metrics analysis, we identify potential entry points for counterproductive practices. 

By taking a proactive stance, we can continuously improve our tactics and make sure that we surpass our reliability goals. Adopting a mindset of constant development, we recognise that addressing anti-patterns entails more than just making corrections—it also entails building a resilient and adaptive environment. This dedication puts us in a position to provide outstanding support and dependability in the rapidly changing world of contemporary technology. 

It’s important to highlight why we are doing this: to make our engineers comfortable and our customers happy with our service. To achieve this, we are focusing on the following areas: 

Incident Management 

Our main goal is to prioritise incident management to prevent and resolve SRE anti-patterns. By taking early measures to avoid future problems, we hope to reduce the number of occurrences that could affect user experience and disrupt services. This strategic move reaffirms our dedication to provide dependable and robust systems while also being in line with industry best practices. Anti-patterns must be effectively managed and mitigated to reduce the frequency and severity of incidents. This proactive strategy improves our systems' overall stability and helps create a more effective incident response structure, which lowers downtime and strengthens our capacity to offer our users uninterrupted, high-quality services. 

Post-mortems, also known as post-incident reviews, are the in-depth examinations carried out following an occurrence. These evaluations are essential for understanding the underlying reasons of accidents, picking up lessons from mistakes, and seeing where improvements may be made. The whole incident response team as well as occasionally other pertinent stakeholders are involved in post-mortems. The intention is to promote a culture of continual development rather than place blame. 

Post-Mortems 

During a post-mortem, teams review the timeline of events leading up to and during the incident, assess the effectiveness of the incident response, and identify contributing factors or anti-patterns that may have played a role. The insights gained from post-mortems inform the refinement of processes, the implementation of preventive measures, and the optimisation of systems to minimise the likelihood of similar incidents in the future. By conducting thorough post-mortems, SRE teams can iteratively enhance their incident management practices, contributing to a more resilient and reliable operational environment. 

Cultivating a Culture of Collaboration 

Dismantling silos and promoting information sharing and shared ownership to promote cross-functional cooperation is essential to cultivate collaboration. It is essential to adopt procedures and technologies that help teams communicate openly with one another. 

Prioritising Effectively 

Effective prioritising is evaluating the risk, assigning error budgets to the most important problems first, and determining how this will affect the overall dependability of the system. Teams can concentrate on the most significant anti-patterns and ensure effective and focused improvements by adopting a user-centric approach and making the most of the tools at their disposal. This prioritisation technique optimises the use of scarce resources, improves user experience, and helps avoid significant disruptions. 

Comprehensive Observability 

Comprehensive observability is a vital aspect of SRE, extending beyond traditional monitoring to include tracing and logging. This three-pronged approach provides SRE teams with a holistic view of system performance. Monitoring detects issues, tracing visualises request paths, and logging captures detailed event data. Together, they empower SREs to swiftly identify and address issues, fostering a culture of continuous improvement and reliability in the dynamic world of technology. Invest in robust monitoring tools and observability practices to gain deep insights into system behavior, allowing proactive responses to potential issues. 

Reliability Framework 

Reliability framework: It detects any problems before they become more serious through proactive monitoring. Error budgets are a notion that aids in allocating resources for countering anti-patterns. Manual error risk is decreased by automation, and failures can be learned from through post-event reviews. By encouraging teams to continuously enhance operational procedures regularly, the framework promotes a culture of continuous improvement. The overall resilience of systems is improved by cross-functional cooperation between development and operations, and by good documentation and information sharing. To put it simply, a dependability framework offers a methodical way to deal with and avoid SRE anti-patterns, making sure that systems adapt to suit shifting requirements. 

Conclusion  

We underwent a strategic transition in our organisational philosophy when we created a new team to focus on SRE. This change emphasised an engineering-centric and proactive approach to managing our technology infrastructure. Inspired by industry best practices modeled by digital giants such as Google, our team is now at the forefront of innovation with a renewed dedication to efficiency, scalability, and dependability. With the ability to use automation, monitoring, and teamwork, SREs can go beyond reactive operations and actively participate in the growth and improvement of our software services. This evolution ensures a more robust and responsive operational framework and puts us in a confident position to handle the challenges of today's dynamic IT landscape. We are excited about the opportunities this change brings and look forward to a future of continued growth and excellence in Site Reliability Engineering.

Watch Ricardo Castro's full presentation here:

Introduction

In the realm of modern business infrastructure, pricing systems serve as the backbone of revenue generation. As user bases expand and real-time data becomes increasingly crucial, the conventional methods of architecture often reveal their limitations. 

Pricing systems have traditionally relied on a client-server polling strategy, where clients repeatedly query the server for updated pricing information. While this approach has served its purpose, it introduces inefficiencies and limitations as the volume of clients and frequency of updates grow. Some of those limitations are the following: 

• Network Congestion and Latency: In a polling architecture, each client sends repeated requests to the server at fixed intervals, regardless of whether there are actual updates to be received. As the number of clients increases, this flood of unnecessary requests can congest the network, leading to increased latency and delayed responses. 
• Limited Real-Time Responsiveness: Polling architectures inherently introduce delays in delivering real-time updates. Since clients can only receive new pricing information when they actively poll the server, there is a delay between the server’s update and the client’s receipt of that update. This lag can be especially problematic in time-sensitive industries, such as financial trading. 
• Scalability Challenges: As the number of clients increases, the burden on the server grows exponentially due to the cumulative effect of numerous polling requests. Scaling such systems to accommodate spikes in demand becomes complex and resource-intensive, often requiring complex load-balancing strategies and additional server provisioning. 

During a recent FanDuel Hackathon, our internal pricing system was revisited to replace its polling strategy by a server-push strategy leveraging the power of GraphQL subscriptions and WebSockets. 

In this article we are going to delve into the skeleton of the solution that was implemented using a demo application to demonstrate the concepts involved. Only the relevant parts of the code will be included in this article but the full project can be found on Github. 

GraphQL 

GraphQL is a query language for APIs. It was developed by Facebook in 2012 and later open-sourced in 2015. Unlike traditional REST APIs, where endpoints dictate the shape and structure of responses, GraphQL enables clients to specify the exact data requirements they have. This client-centric approach allows for a more efficient data retrieval process, reducing the amount of unnecessary data transferred over the network. With GraphQL, the server responds with precisely the requested data, eliminating the need for multiple round-trips and streamlining the client-server interaction. 

WebSockets 

WebSockets is a communication protocol that provides a full-duplex, bidirectional communication channel over a single, long-lived connection between a client and a server. Unlike traditional HTTP communication, which involves sending requests from clients and receiving responses from servers, WebSockets allow both clients and servers to send data to each other independently, without the overhead of creating new connections for each interaction. 

GraphQL Subscriptions 

One of the less well-known features of GraphQL is the GraphQL subscriptions. This feature extends the capabilities of the GraphQL query language to enable real-time data updates going from the server to the client. While standard GraphQL queries and mutations are designed for fetching and modifying data, subscriptions are tailored for scenarios where clients need to receive live updates when specific data changes on the server. 

One of the possible ways to implement GraphQL subscriptions is to rely on the WebSockets protocol to propagate the data from the server to the client when necessary. This is what we did to scale our internal pricing system and we will now go in detail into the technical solution.

Application Architecture and Implementation Steps 

We are going through the solution that was implemented and dissect it to explain the concepts that allowed us to scale our pricing system. 

1. The first thing that needed to be done was to specify the schema to be used by the GraphQL engine: 

schema { 
    query: Query # Schemas must have at least a query root type 
    subscription : Subscription 
} 
 
type Query { 
    dummyQueryValue : String 
} 
 
type Subscription { 
    marketPrices(marketIds:[String]!) : MarketPrice! 
} 
 
type MarketPrice { 
    id : String 
    name : String 
    price : Float 
}

Here, it was specified a GraphQL subscription for market prices notifications. Using that subscription, it is possible to subscribe for prices notifications of specific markets using their IDs. It is also possible to select the information we want to get in each notification from the list of available parameters - ID, name and price. Note that a real pricing system would probably provide a lot more information, but we are going to keep it simple for demonstration purposes. Note also that we needed to specify a Query type at the root of the schema even though we did not need it - this is a requirement from the GraphQL engine which requires us to define a top-level query type. Learn more about the syntax of the GraphQL schemas. 

2. Then, we created a Java class to map the GraphQL MarketPrice type into a Java object. Note that the names of the fields in the Java class need to match the names defined in the schema type so that the GraphQL library is able to properly do the required mapping. 

public record MarketPrice(String id, String name, double price) {}

3. After creating a Java class, we needed to implement a WebSocket server to support the GraphQL subscriptions. For that purpose, we have used the spring-boot-starter-websocket dependency that allows to very easily setup a WebSocket server by creating a Spring configuration class that extends the WebSocketConfigurerclass and that it is enriched with the @EnableWebSocket annotation. Then, we registered the WebSocket handler for GraphQL to process requests made to the /ws/graphql path. 

@Configuration 
@EnableWebSocket 
public class PricingNotificationsApplicationConfig implements WebSocketConfigurer { 
 
	@Override 
	public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) { 
		GraphQL graphQL = GraphQLMarketPricesInitializer.build( 
				new MarketPricesDummyPublisher(), 
				"graphql/market_price.graphql"); 
		registry.addHandler(new GraphQLWebsocketHandler(graphQL), "/ws/graphql"); 
	} 
}

4. For the GraphQL WebSocket handler, we needed to create a GraphQL instance capable of receiving the GraphQL market prices subscriptions and replying with the desired information. The GraphQLMarketPricesInitializer class is responsible for building that instance. It first reads the GraphQL schema exposed above and parses it:  

public class GraphQLMarketPricesInitializer { 
  ... 
  public static GraphQL build(MarketPricesPublisher marketPricesPublisher, String graphQlSchemaFile) { 
    InputStream graphQlSchemaStream = GraphQLMarketPricesInitializer.class.getClassLoader().getResourceAsStream(graphQlSchemaFile); 
    Reader graphQlSchemaReader = new InputStreamReader(graphQlSchemaStream); 
    TypeDefinitionRegistry typeRegistry = new SchemaParser().parse(graphQlSchemaReader); 
    ... 
  } 
  ... 
}

5. Then, the GraphQLMarketPricesInitializer class does the required wiring using the GraphQL library to link the marketPrices subscription to the appropriate handler which should be an implementation of the GraphQL library DataFetcherinterface. To handle the market prices subscriptions, we’ve created a DataFetcher that uses Reactive Streams to create a stream of price notifications for the desired markets. Note that the names used here to do the wiring must match the names specified on the GraphQL schema file.

public class GraphQLMarketPricesInitializer { 
 
  private static final String SUBSCRIPTION_WIRING = "Subscription"; 
  private static final String MARKET_PRICES_SUBSCRIPTION = "marketPrices"; 
  private static final String MARKET_PRICES_SUBSCRIPTION_MARKET_IDS = "marketIds"; 
 
  public static GraphQL build(MarketPricesPublisher marketPricesPublisher, String graphQlSchemaFile) { 
    ... 
    RuntimeWiring wiring = RuntimeWiring.newRuntimeWiring() 
            .type(TypeRuntimeWiring 
                    .newTypeWiring(SUBSCRIPTION_WIRING) 
                    .dataFetcher(MARKET_PRICES_SUBSCRIPTION, marketPricesSubscriptionFetcher(marketPricesPublisher)) 
            ) 
            .build(); 
 
    GraphQLSchema schema = new SchemaGenerator().makeExecutableSchema(typeRegistry, wiring); 
    return GraphQL.newGraphQL(schema).build(); 
  } 
 
  private static DataFetcher<Publisher<MarketPrice>> marketPricesSubscriptionFetcher(MarketPricesPublisher marketPricesPublisher) { 
      return environment -> { 
          Set<String> marketIds = Set.copyOf(environment.getArgument(MARKET_PRICES_SUBSCRIPTION_MARKET_IDS)); 
          return marketPricesPublisher.getPublisher(marketIds); 
      }; 
  } 
}

6. After implementing the GraphQL WebSocket handler, we needed to create a continuous stream of market price notifications. For that purpose, we’ve created the DummyMarketPricesPublisherclass using Reactor, which is an implementation of the Reactive Streams specification. In the DummyMarketPricesPublisher implementation, we’ve defined a set of dummy markets available for subscription and then we’ve created a continuous stream of market prices notifications that emits new market prices for each dummy market specified above with a periodicity of 1 second.

public interface MarketPricesPublisher { 
    Publisher<MarketPrice> getPublisher(Set<String> marketIds); 
} 
 
public class MarketPricesDummyPublisher implements MarketPricesPublisher { 
  private static final int PRICE_UPDATE_INTERVAL_SECONDS = 1; 
 
  private record DummyMarket(String id, String name) {} 
 
  private final List<DummyMarket> dummyMarkets = List.of( 
          new DummyMarket("1", "Porto vs Benfica"), 
          new DummyMarket("2", "Liverpool vs Manchester United"), 
          new DummyMarket("3", "Braga vs Madrid"), 
          new DummyMarket("4", "Ajax vs Barcelona"), 
          new DummyMarket("5", "Arsenal vs Milan") 
  ); 
 
  public MarketPricesDummyPublisher() { 
      publisher = Flux.interval(Duration.ofSeconds(PRICE_UPDATE_INTERVAL_SECONDS)) 
              .flatMapIterable(value -> dummyMarkets.stream() 
                      .map(dummyMarket -> new MarketPrice(dummyMarket.id, dummyMarket.name, calculateRandomPrice())) 
                      .collect(Collectors.toList()) 
              ); 
  } 
  ... 
}

7. Then, the last step was to handle incoming WebSockets connections and provide the market price updates for the desired markets. For that purpose, we’ve created a class named GraphQLWebsocketHandler that implements the WebSocketHandler interface. This is the class that connects each WebSocket connection to a Reactor stream of market price updates for the requested markets and that leverages the GraphQL toSpecification() method to return only the requested data.

public class GraphQLWebsocketHandler implements WebSocketHandler { 
    ... 
    @Override 
    public void handleMessage(WebSocketSession session, WebSocketMessage<?> message) { 
        handleNewGraphQlSubscription(session, ((TextMessage) message).getPayload()); 
    } 
    ... 
    private void handleNewGraphQlSubscription(WebSocketSession session, String message) { 
        ExecutionResult executionResult = graphQL.execute(ExecutionInput.newExecutionInput().query(message)); 
        SubscriptionPublisher subscriptionPublisher = executionResult.getData(); 
        Flux.from(subscriptionPublisher) 
                .takeWhile(ignored -> session.isOpen()) 
                .subscribe(marketPrice -> { 
                    try { 
                        session.sendMessage(new TextMessage(marketPrice.toSpecification().toString())); 
                    } catch (IOException e) { 
                        throw new RuntimeException(e); 
                    } 
                }); 
    } 
}

The image below shows the final result: in the left panel, you can see the clients making the subscriptions (subscription { marketPrices(marketIds: ["1","3","5"]) { id price } }and subscription { marketPrices(marketIds: ["2","4"]) { id price } }) and getting the desired information. On the right side you can see the server receiving and accepting those subscriptions:

We can then leverage the power of GraphQL to request extra data that we may want, such as the name of the market - subscription { marketPrices(marketIds: ["2"]) { price name } }:  

With this solution, we can remove the necessity for constant server polling, and we can select the exact data that we want to receive, which ultimately facilitates the scaling of the system. 

Conclusion 

To sum up, the adoption of GraphQL subscriptions in place of a client-server polling approach has been a critical change that has improved system efficiency and scalability. With this change, our system can now handle increased traffic and provide real-time updates, guaranteeing that our pricing information is dynamic, correct and current. We have enabled our system to easily manage growing loads, which greatly enhanced the user experience, minimised network overhead, and optimised our data exchange. 

We've simplified our codebase, making it easier to maintain and modify, by using this contemporary methodology. This change is a calculated attempt to future-proof our pricing structure so that it can adapt to changing market conditions and yet be flexible and responsive. 

The implementation of GraphQL subscriptions have clearly shown their capacity to improve the functionality and dependability of our pricing system, while also laying the groundwork for a more responsive pricing structure that is future-ready, as we continue to embrace cutting-edge tactics and technologies.  

Check out the Backend Developer opportunities we have available and join us!

Betfair and Paddy Power, the betting and gaming platforms part of Blip and the Flutter Entertainment Group, recently underwent a major overhaul by transitioning from AngularJS to ReactJS. This transition aims to bring together into a single experience the different products, with the main goal of delivering world-class applications that provide the safest betting and gaming experiences across the industry. 

Problem statement and rationale for maintainable codebases

The ability of our applications to meet the needs of our customers and have a well-defined, maintainable architecture is crucial for the business. Architecture and codebases begin to deteriorate over time, which causes a slowdown in the development of new features, slower applications with performance issues, security flaws, and other problems. 

Evolutionary architecture 

Being able to change and evolve our applications gradually is key for our business. We need to keep up with the industry to accommodate a constant stream of innovation in terms of tools, frameworks, and techniques because the development ecosystem is constantly changing.

Throughout this article, we will explore the journeys taken by Paddy Power and Betfair to transform our technology stacks within the customer-facing set of applications of the brands. 

As João Ferreira (Senior Front-End Developer from Paddy Power) points out:  
“There are no absolutes or a universal migration guide to follow when it comes to transition from AngularJS to ReactJS. It all depends on the way the apps are built in Angular.” 

From AngularJS to ReactJS - The need for change 

This decision was driven by the need for a more flexible and scalable front-end framework that could handle the complex requirements of the platform.  

As the AngularJS framework reached end-of-life status in 2018, our developers searched for alternatives to decommission it and were faced with a decision to be made around evolving frontend component architecture. ReactJS was ultimately chosen as the more modern and future-proof UI library to replace AngularJS as part of the technology transformation within our customer-facing applications.  

We had to decide whether to completely rewrite or refactor a codebase to make this migration when it came to the strategy for evolving the UI components. At Betfair, it was decided to move forward with a new set of features and an improved user experience for the existing products, while at Paddy Power, it was decided to use an evolutionary approach by refactoring existing components. 

Rewriting vs Refactoring

The best approach for a codebase that needs to evolve — to rewrite or to refactor — is a topic of intense debate within the industry. It all comes down to making the codebases cleaner, more maintainable, and future-proof. Which approach is best depends on the circumstances of each individual project. There are always trade-offs when it comes to software architecture, and this was no different. 

Rewriting

The transition process involved rewriting an application highly coupled to AngularJS into a framework agnostic application with ReactJS at the UI layer, ensuring seamless integration with the existing backend infrastructure. 

The upsides

• Clean slate: Rewriting gives developers a chance to start over with a new codebase, which can result in a system that is cleaner, better organised, and easier to maintain.   
• Updates to technology: Rewriting makes it possible to use the most recent frameworks, programming languages, and technologies, which can enhance performance, security, and maintainability.   
• Eliminating technical debt: A rewrite offers the chance to eliminate accumulated technical debt, creating a more reliable and stable system.   
• Better performance and user experience: Because developers can more easily add new features and address old problems, a well-done rewrite can result in better performance and user experience. 

The downsides  

• Time and cost: Rewriting typically takes longer and costs more than refactoring. It could divert resources away from other active projects because it necessitates a significant amount of development, testing, and deployment.   
• Failure probability: Rewriting carries a certain amount of risk, and there's a possibility that the new system will be less stable and functional than the original. It might also bring about new bugs and problems.   
• Inconvenience to users: Users may experience inconvenience from a rewrite, particularly if the software or service is widely used. Users might need some time to get used to the new system, and some features they were used to using might be temporarily unavailable.

Refactoring

Refactoring means restructuring code while not changing its original functionality. It entails rethinking and restructuring the codebase to conform to ReactJS's architecture and best practices. 

The upsides  

• Gradual changes: Refactoring enables developers to incrementally alter an existing codebase, enhancing the structure and maintainability over time. Since a full rebuild is not necessary, it can be done gradually and with less disruption to users.   
• Maintaining functionality: Refactoring aims to enhance existing code while maintaining its usability. Therefore, there won't be any significant interruptions to users' ability to use the software.   
• Risk reduction: Refactoring generally carries a lower risk of introducing new bugs or breaking existing functionality than rewriting because it is done in small, manageable steps.  
• Efficiency: Because refactoring builds on existing code and concentrates on targeted improvements, it can be more time and money efficient.   
• Domain knowledge retention: When working on refactoring, developers typically have a solid grasp of the current system and its domain, which aids them in making better design choices. 

The downsides

Lessons learned

• Separate business logic into components independent of frameworks: Assuring that business logic and the rendering layer are separated is key to ensuring a good separation of concerns and that those two core aspects of an application can have proper maintenance.  
• Build for evolution and change: Make things as dynamic as possible by integrating content management systems, experimentation and multi-variant testing tools.  
• Backend for Front-end pattern: Leverage Backend for Front-end (BFF) patterns and serverless for abstracting logic from the client code. 
• State management: Leverage state management libraries to abstract the UI layer from the business logic. Tools like Redux and Apollo were also added during the migration as it helped to manage state within the applications in an effective manner. 
• Design system: Leverage design system best practices for composability and adoption of design tokens is crucial. 
• Multiplatform support: We can cut down on the amount of code needed to build our applications by targeting to develop once and reuse on multiple platforms. Code reuse between native platforms can be facilitated by tools like React Native. Other tools that enable code to run on various platforms, including on the server-side, like Web Assembly, can also add great value. 

Final thoughts 

Overall, switching from AngularJS to ReactJS gives developers a more effective, adaptable, and maintainable frontend development method, improving the overall calibre and user experience of web applications.  

This transition has encouraged developers to embrace change and take advantage of the opportunities for growth and improvement. There is always a learning curve when switching from one framework to another. Everything has trade-offs, like most architectural decisions, but we should never lose sight of the goal that is to enhance the customer experience.  

With the migration, besides learning that it is important to refactor frequently and early, we have learned that sometimes rewriting small components from scratch might be more beneficial than maintaining them in the long run. We have also learned how crucial it is to align the product management, UX/UI and development teams for better decision-making and to create the best-in-class development experience for fast-paced shipping culture. 

Are you a frontend developer and this article instigated you to learn more about how we do frontend at Blip? We have the perfect opportunity for you then. Join us! 

The success of any project hinges on the productivity and satisfaction of its developers: if they are not productive and satisfied, the whole SDLC (Software Development Life Cycle) is doomed to fail. At Blip and in the Flutter group, we are constantly trying to improve our ways of working to ensure we deliver the best Developer Experience (DevXP) possible to our people. Join us in this article as we delve into the main improvements that we are currently working on to unleash the power of seamless collaboration.

Learning how we can uplift our DevXP to unlock the full potential of our development teams in the ever-evolving world of software development is the first step to achieving a seamless collaboration between developers, tools, and processes that pave the way for a new era of software development excellence. 

Developer Experience goes beyond the mere availability of programming languages and frameworks. It acknowledges the value of creating a setting where developers can flourish, innovate, and collaborate with ease. By prioritising the Developer Experience, we can increase productivity and produce superior software solutions to meet the constantly expanding demands of the modern digital world. This is especially relevant in organisations like ours, where we have several different teams and brands united for the same goal of building the best products, as is the case of the Global Betting Platform (gbp).  

Active Listening to the Community Feedback 

When the goals are great, so are the challenges – and sometimes some changes in our processes are necessary so that we can achieve excellence. These changes may involve adapting our processes, procedures and even learning how to collaborate in a different way. To be able to know what to change, we need to be open to feedback, listen to what the community has to say and investigate pain points. 

The challenges raised by the community feedback that affected the development life cycle and developer experience included: 

• Cross-division tooling dependency; 
• Code-reviews bottlenecks; 
• Lack of application versioning policies; 
• Poor visibility of breaking changes; 
• Lack of standard testing documentation. 

Starting the Improvement Journey 

Proof of concept (POC) workstreams called "Developer Experience Uplift" were started to address the community's feedback. These workstreams sought to pinpoint the underlying causes of the major issues, identify potential technical fixes, outline an ideal target state, and offer high-level estimates of the required work. These workstreams concentrated on the following areas of analysis: 

• Continuous Integration (CI) pipeline 
• Standardised approach to testing 
• Branching strategy 

 The ultimate objective of these workstreams was to enhance the developer experience overall by enabling developers to create software more quickly, while enhancing productivity and developers' well-being. The feedback from the community was crucial in identifying areas for improvement and set us on a quest to make relevant improvements, with the ambition to have a more streamlined and efficient development process. 

Solving Bottlenecks 

During the POC, we have come to understand the source of the problems each workstream faced, and we have studied possible solutions to solve them: 

• CI Uplift: All CI pipeline was implemented in the heritage Jenkins instance, which caused dependencies on the heritage artefact repository manager. Being the target state to achieve divisional autonomy and prompt and automated feedback on proposed changes, this current divisional dependency could be overcome by implementing the CI pipeline on GitHub Actions and transforming the heritage artefact repository manager into a shared asset. Using GitHub repositories may reduce friction with these internal dependencies. 
• Testing: Since most tests depend on the heritage infrastructure to run, a possible solution would be that no component tests depend on the heritage infrastructure and component tests run on GitHub Actions. This would solve the divisional dependency, providing divisional autonomy and prompt and automated feedback on proposed changes (tests). 
• Branching Strategy: Currently, release versions are based on Jenkins build numbers without a semantic meaning and an informal Gitflow branching model is being used. This lack of standards problem could be solved by adopting semantic versioning (through SemVer specification) for more meaningful version numbers and to formalise the branching model to Gitflow. This way, we could achieve a set of policies and conventions for a standard application versioning and documentation. 

How we Are Transforming the Product Across Divisions 

After the POC findings, we have decided on a set of changes so that we can achieve the desired outcomes by implementing the Developer Experience Uplift initiative. It is important to note that the implementation strategy may need to be adjusted based on any unforeseen challenges or obstacles that arise during the process. Additionally, regular monitoring and evaluation of the initiative's progress will be necessary to ensure its success. The implementation strategy has the following goals and dependencies: 

Getting Rid of Barriers 

By enhancing the way engineers use Git branching, we hope to improve the experience of developers who contribute to gbp. By implementing a multi-team branching model, we can have a standard branching workflow (common SDLC per repository) and a standard versioning format. We were able to determine that the primary gbp Git workflow, Feature Branch, is not robust and flexible enough to gbp's current reality. Gitflow branching's implementation makes it simple and straightforward for teams from various divisions to collaborate. 

Removing Dependencies 

By using Docker to containerise testing dependencies, we are removing the reliance on the heritage infrastructure and, while at it, removing any dependency from the different UK&I brands and we are fostering testing autonomy. In this manner, we can run testing frameworks separately from divisional tooling in a stable environment. New features can be tested without affecting other divisions thanks to the setup of testing processes and tools. Furthermore, by enabling testing autonomy, we are increasing the speed and efficiency of our testing processes. This allows us to quickly identify and resolve any issues, ultimately leading to a higher-quality product for our customers. 

Resolving Access Problems 

Moving the CI pipeline to GitHub so that all engineers can have access enables: 

• Automated unit and component tests feedback on the pull requests; 
• Automated changelog and release notes; 
• Additional automated feedback (security, SDLC policies).  

Each pull request's automatic feedback creation ensures a higher level of consistency, helping to reduce complexity and provide more context for reviewers. Furthermore, the automated feedback system allows for quicker identification and resolution of issues, as well as increased efficiency in the development process. With these features in place, teams can better collaborate seamlessly and ensure that code changes are thoroughly tested and reviewed before being merged into the main branch.  

Improving Testing Documentation 

Test documentation can be made readily accessible to all divisions as GitHub files, by condensing it into Markdown files on GitHub. Having the required documentation makes it possible to confidently comprehend, design, and execute tests. Moreover, these practices allow us to keep testing documentation up to date, ensuring all team members are on the same page and can effectively collaborate on testing efforts. This ultimately leads to a more efficient and effective testing process.  

Knowing the Risks 

As any project entailing a major transformation, there are risks that need to be managed and challenges that need to be overcome, namely: 

• Modifications to the SDLC process and tooling may bring on resistance. There is a small risk that the changes will impact current pipelines. It is suggested that teams first test all changes (branching strategy + pipeline addition) in a clone repository before switching over. 
• Estimating efforts may be impacted by variations in test automation suite size and complexity of different services and capabilities. CI on GitHub needs to be completely migrated to have full value delivered. 
• The full CI value depends on the accomplishment of the other objectives (multi-team branching and testing autonomy). Accurate branching standardisation is essential to CI.  
• SemVer was outside the POC's purview, and there are still more unanswered questions. A strong alignment between divisions is necessary for the SemVer migration to work. A partial transition to SemVer with only 5 gbp services adopting it might not have the desired effect. 
• There is a risk related to the effort coordination across the different divisions/brands. As we worked in a shared codebase model, these changes need to be coordinated across everyone contributing and any misalignment might lead to overwork. 

Where We Stand and What We Have Learned So Far 

It is major for us to actively seek and address the challenges that come our way, in order to improve the overall efficiency and effectiveness of our work. By evaluating our bottlenecks, designing, and implementing solutions, and continuously evaluating and adapting processes, our teams can successfully overcome these obstacles and achieve greater success. As we live and learn, while it is important to remember that these adjustments are a normal part of the process and can ultimately lead to greater success as a group, communication channels must be kept open so we can continuously know the team's feel and adjust as we walk together on this journey to build great software for our customers. 

If you enjoyed learning more about our developer experience at Blip thus far, check out the job openings we have available and join us!

Since the beginning of the millennium, software development teams started working on a series of values and principles named agile methodologies. For a while, the world lived under the belief that these methodologies were exclusive for them.

Let’s picture a treasure map where we have a starting point and a X that marks the spot where the gold is hidden. We want to reach for the gold in the fastest and safest way. We can trace a path and force ourselves to follow – from point A to B, B to C and so on (Waterfall method). But what if on our traced path we find a magic carpet? Should we use it and shorten the distance or get stuck with the idea and the process? What if it rains and the bridge we need gets ruined, do we stay powerless or do we find an alternative path and build a new bridge?

Are the development teams the only ones in search for the gold? In these teams the idea of an Agile methodology is to speed up deliveries, correct and detect problems more quickly. Reviewing plan, not being bound by processes or procedures, but rather the autonomy of the team and their need to meet a goal. This gives an empowerment to individual autonomy, individual creativity, and an idea of community where interactions are valuable rather than a structures process written in stone.

I believe that this idea of working in an agile way not only can but must be adapted to the reality of an HR professional. We should look at it as a foundation and allow HR teams to adapt them and create ceremonies that adopt the same principles. Limiting ourselves by looking at these methodologies as if they are constant and could not be adapted to other realities, well, is not being agile.

To give an insight on how to do this, let’s imagine a Talent Acquisition team that aims to double the number of working teams in the company which it is part of. How can this team become agile? The first step is to look at the end goal, for example hiring 80 developers in a year, and break it into smaller pieces, set small goals and tasks that get us to the treasure.

We start the planning phase here. We will divide this chunk into tasks and distribute them to the team. In this planning, as a team we will look at which roles comprise the first batch, we will understand and analyse recruitment patterns, response rates on LinkedIn, application rates, understand timings that each task will take up. We’ll also look at the time of year and validate if there are bank holidays in these weeks or if any team member has booked holidays… It is important that the plan is ambitious but realistic. Once the tasks have been aligned it is up to us to distribute them and it is not agile at all to think of this in an egalitarian way, i.e., twenty tasks to be divided by four team members. We will make an equitable distribution according to the effort and time we estimate each task would take.

Involving the teams in the planning of the tasks helps to create a sense of commitment to the goal and to have a 360º view of the needs and what will be everyone's goal rather than an individual goal. It is a way to ensure that everyone's concerns are heard and to anticipate some problems that we think may arise along the way.

After planning we start our sprint. It is common for development teams to establish 2-week sprints, however, in the case of a TA team, it seems easier to think in 1-month sprints considering the metrics and the time that recruitment processes usually take, since many of the processes have more than one phase and we can consider that a process could be started and finished in one month.

The tasks of a TA team are more than just closing positions. Different metrics can be created: send 100 messages on LinkedIn per week, put 10 candidates in process, improve the recruitment process, conduct 12 interviews per week, share 2 job advertisements per week, write articles on LinkedIn.

Another moment we must give importance to, besides planning, is the daily status meeting. The team's day must start with a short meeting where we share the status of our tasks, considering the same example above - how many interviews have been done, progress of candidates who are in the pipeline, difficulties or delays that may have happened in the process. This way, the team is able to track the progress as a whole and continue to have the peripheral view on the path to our end goal.

When we reach the end of our first goal, we should look back. That is, once again as a team we should get together and understand what difficulties we encountered and create improvement actions, which processes went well and must be replicated in the future. To simplify, divide a whiteboard into two parts: on one side we’ll have what went well, and on the other, points to be improved. Each element is free to add whatever they want and find relevant in those columns (you can do it with post-its, on Zoom whiteboards, in Excel, on a Trello screen). At this stage we can't expect to be discussing as a team half a day - we all know we would easily get lost, and the aim is to focus on what is the most important. In this case, I suggest that after everyone has given their input, each team member can vote for three points per column on what they consider the most important. The most voted topics will be the ones considered priority and this is where the team will apply their time. Improvement must be seen as a continuous process; we may not go through all the topics, but we will pay attention to the priorities and keep the focus.

Once this stage is complete, we return to our starting point: the next planning. We can look at what tasks we did not successfully complete, and which have been carried over to this planning, as well as those that we can fit in without having a negative impact on the team's routine. We must prioritize the tasks and see if it is feasible to fit them in here or if it is more strategic to leave something for the next planning due to the impact it has on the team members' time and priority.

Throughout this year I worked with a team that tried to find its way within Agile methodologies and create these moments. Besides believing that we improved the path to achieve our goals, we learned to work better as a team, we were able to be more consistent in our deliveries and manage our workload better.

Introduction

Web development is constantly evolving, with innovative approaches and technologies shaping how we build and maintain applications. One such approach that has gained significant attention among frontend developers is the use of micro-frontends.

Micro-frontends involve breaking down large, monolithic frontends into smaller, more manageable components. Each micro-frontend is an independent, self-contained unit of code that handles a specific responsibility within the overall application, often corresponding to a distinct section of a webpage.

The advantage of micro-frontends is their autonomy. These components can be developed, deployed, and maintained separately, enabling faster development cycles and easier updates. By communicating through APIs, micro-frontends can interact seamlessly while being built with a variety of technologies and frameworks, giving teams the flexibility to choose the best tools for each part of their application.

How we are using micro-frontends to accelerate our business 

Blip, as part of the Flutter Entertainment Group, is looking to adopt micro-frontend technology across all sports platforms. As a global group, we have several distinct products that need a consistent design interface, way of working and user experience. With this in mind, we want multiple teams to contribute faster and to be able to reuse content while building the same apps, so that what changes is configuration, not code. This allows, for example, to reuse events search UI instead of fully developing it each time. 

What is so good about it? 

Easier coordination among teams 

Coordinating and delivering updates simultaneously can be challenging when different teams are working on distinct features for the same product. With micro-frontends, each team can manage their deployment process without affecting other teams who might be testing out experimental code concurrently. This makes it possible to carry out releases and updates with features that are already live in a more planned and effective manner. 

More time to build new features 

As micro-frontends allow code reusability and sharing, teams can focus on creating new features instead of duplicating what has already been done every time it is necessary to implement a common functionality. This allows faster feature rollout. 

Flexibility and independence 

This kind of architecture for creating frontend apps gives frontend teams the same flexibility and speed that microservices give backend teams. This happens by segmenting a web application into several modules or individual functions and implementing them independently. This independence allows teams to work on their components without depending on others and stops tight coupling. 

Easier maintenance and testing 

As micro-frontends reduce duplication across applications, they are easier to maintain long term when changes need to be made. They allow developers to work with smaller and more manageable apps, which reduces testing time. 

More efficient performance 

To load all elements of a page, traditional web development frequently needs several requests. This may not be efficient if some parts are not required for rendering some pages. To improve performance, it is possible to postpone loading these additional modules until they are needed by using micro-frontends. At the same time, using micro-frontends increases fault tolerance since the system is not ruined when one of the components fails. 

Connection to backend 

The host app allows connection to the backend as it can provide a set of capabilities that the component apps can use. This is another advantage of the micro-frontend's architecture since it is possible, for example, to handle backend streams to provide live data. 

Principles that should be followed when using micro-frontends in UI development: 

• One host app for multiple component apps; 
• Minimal dependency: development, infrastructure and deployment processes are independent. There are only dependencies in the runtime, which is inevitable if the apps need to communicate with each other; 
• The host app is a distinct, standalone app that orchestrates its children, so it should be built and maintained independently. Only standard tools should be used so that developer churn does not affect maintenance long term; 
• The team creating the component (children) app should be free to use any setup and/or framework technology they choose, entirely independent from the host app; 
• Component apps should be allowed to use their framework version and not depend on the host version;  
• To integrate apps in the frontend without having to go through backend for simple communications, a message-based communication mechanism needs to be implemented between the host and the component apps. 

Micro-frontend using iFrames 

iFrames (Inline Frames) are an HTML element that allows embedding one HTML document within another. This structure enables the display of multiple applications or content within a single browser window, effectively creating a page composed of distinct, self-contained sub-pages. Each sub-page can be independently managed, often by different teams or technologies, making iFrames an ideal solution for micro-frontend architectures.

At Flutter, we've determined that iFrames meet all the critical technical requirements for implementing micro-frontends. They are a well-established solution, with their native technology operating independently from component apps. The integration via the Window Messaging Web API ensures minimal dependencies, allowing for flexibility and ease of transition should we choose to adopt a different solution in the future.

How are the host app and its child applications integrated? 

It is required a configuration file with child app definitions determining their URL to load the child app bundles into iFrames. The micro-frontend lifecycle is a strategy for negotiating child apps mounting onto the main window for actual use. It is the cycle any micro-frontend component app needs to undergo during its life inside the micro-frontend host to be properly integrated into it. This is a standard protocol that all component apps need to obey, and the host needs to honour. The host app first renders an iFrame that loads the child app in a 3-step protocol that leverages the inter-app messaging system: 

1. The child app (running in its iFrame) messages the host to inform it is now mounting. 
2. The host app replies with authentication/authorisation data.
3. Based on the auth/auth data it received, the child app replies with:
   a) Mounted: the child app is now part of the micro-frontend app and can be used.
   b) Unmounted: the child app refuses to mount, and it is expelled from the host.

Implementation status

At Flutter, we're currently in the experimentation and learning phase of implementing micro-frontends. This approach, which involves breaking down large, monolithic frontends into smaller, self-contained components, holds great potential for improving the scalability, flexibility, and maintainability of our applications. During this phase, we're exploring how best to structure and integrate these independent components, ensuring they communicate effectively through APIs and work seamlessly across various technologies and frameworks.

Once we've refined our approach and identified the optimal strategies for our teams, we’re excited to leverage the full benefits of micro-frontends—accelerating development, enabling more efficient collaboration, and ultimately creating a more agile and robust frontend architecture.

Introduction

Blip is part of the Flutter Tech group, a global organisation that provides excitement and entertainment in a safe and responsible way to 13 million customers, spread over 100 countries. Our people across 15 locations contribute to Flutter being a global leader in sports betting and gaming industry. We are the number one digital sportsbook in the world, with annual revenues of over £4bn and a portfolio of 14 market-leading brands like FanDuel, PokerStars, Betfair, PaddyPower, Sky Betting and Gaming, among others. Each brand has best in class products, powered by our in-house technology.

Why Inner Source?

Producing high-quality products requires a lot of effort and complexity. For this reason, creating separate products for each brand would mean getting little value from operating as a Flutter group. Instead, we use Inner Source to share the effort to build some of our best and most complex products across several brands. Considering this level of impact and dimension, Flutter recognised an opportunity to leverage each brand's value. This means:

• Reusability of products and features to save time, resources, and development costs to achieve faster time to market;
• Boost collaboration worldwide by leveraging expertise, innovation and improving code quality;
• Enhancing scalability and resilience so that we can use the investment to roll out features to other divisions;
• Cultivate intrinsic motivations by building a community as large as the organisation’s entire development staff;
• Adding value at scale and speed to the organisation’s development approach.

This is where Inner Source comes through as the desired development method: Inner Source integrates open-source processes and collaboration techniques into the organisation framework, comprising transparency, collaboration, and alignment as the main core values.

Rather than a methodology, it is a philosophy and culture designed for code, documentation, and other artefacts to be publicly accessible internally, so that anyone in the organisation can contribute to a common platform with shared codebases. It is called inner source because the software is sourced internally – just like open source, but inside the organisation.

How Do We Make It Possible?

Any internal team can use the Flutter group's Inner Source products, and any of our 4000 engineers from any division or location are welcome to contribute. We establish the ideal conditions for an effective cross-brand cooperation by:

• Determining the appropriate levels of accountability among those contributing and maintaining the product at its best;
• Increasing visibility of all the work planned and already completed for that product;
• Encouraging the necessary community alignment through the creation of communication channels, events, and artefacts;
• Breaking down the product to a more modular approach (e.g., capabilities that can deliver business value on their own).

Conclusion

More than a well-defined approach and tailored governance model, implementing Inner Source throughout such a large-scale group is a long journey and requires a cultural shift. Still, we believe that the benefits and impactful outcomes of its implementation make up for it.

Learn more about Inner Source.

Think about a recent decision you’ve made. What was that decision based on? Your experience? Maybe. Your context? Sure thing. Data? Definitely!

At Blip we are no different. Just like you, our brands and platforms rely on a huge (really, it’s insane) amount of data that is collected 24/7 with the goal of providing our customers with a better overall and personalized experience. In 2021, all our brands combined had an average of 4.5 million monthly players and while they see odds, lines, spreads, play-by-play updates, deposits, and withdrawals, we only see data, data, and data. Each game generates tens of millions of individual data points that must be collected, combined, understood, and acted on and this is what we do.

Our data teams

At Blip, we are creating two new teams to work alongside the US and UK teams. They will be focused on supporting our FanDuel brand by building automation tools to organize and model our data so that we can provide a common source of “truth” that will allow us to make decisions in a smarter and faster way. Moreover, these teams will also be responsible for our own Data warehouse, and they will work with the most recent technologies to optimize and improve it.

We’ll be designing highly scalable and available systems capable of running hundreds of data pipelines at the same time, we’ll be processing huge workloads using distributed data systems and we’ll also use Kafka and Kinesis to process real-time streaming workloads with distributed event systems, but you know what’s more important than all this? We’ll be taking on a completely new challenge for Blip that will allow us to learn more each day and keep growing.

Transactional online businesses are prime targets for hacker activity. Mostly for profit, but also to disrupt the markets those businesses operate on, among many other diverse crimes, malicious players and automated bots continuously prod Flutter’s brands online surface looking for a single weakness they can abuse.

To protect its clients and assets, Flutter has multiple information security teams distributed across its locations. Blip’s office in Porto is the main location for two: the Information Security Engineering team, that builds and manages our security tooling, and the Security Testing team, which provides penetration testing, red teaming and general vulnerability detection capabilities for our online estate.

One of the tasks we (the Security Testing team) are best known for internally is the management of Flutter’s Bug Bounty program, started by us in 2018 and managed through the HackerOne (H1) platform. Before 2018, if a hacker were to find a vulnerability in one of our products, they would report it to us by email, a process that would not scale and was difficult to manage. To improve it we decided to begin a formal Bug Bounty program. We evaluated several online bug bounty platforms and eventually selected H1.

HackerOne enabled us to attract top talent among security researchers and maintain a communication channel with them through the platform. At the same time, it allowed us to establish clear rules of engagement for researchers, and it supported the management of each reported vulnerability, from triage to closure. Significantly, H1 also supports monetary bounties, so we can reward the work of security researchers and support their efforts on responsible disclosure.

In 2018 we started a small, private program, inviting 20 of the best H1 researchers to test our main websites. During the next four years, we gradually grew this program into a fully open Bug Bounty. Today, the program encompasses the online surface of two of Flutter’s major brands. We offer a maximum of $3000 per valid report, and anyone can report a vulnerability and be eligible for a bounty. So far, we collaborated with over 250 hackers who have made a positive contribution to the maturity of our security posture and the confidence we have on our security controls.

Successfully running a bug bounty program is a communal effort, with ramifications that reach way beyond the global Flutter security team. When a vulnerability is reported to us, we work with our SOC and the on-call personnel to triage reported issues and respond in the right timeframe to valid reports. Occasionally we have received reports of some unexpected vulnerabilities, from a zero-day RCE (CVE-2019-19781) we reported to Citrix, to complex chained vulnerabilities, so once we triage the vulnerability as valid, we pass it onto the relevant teams, working together with them to understand the issue, identify any impact, and establish how it can be fixed.

In the end, the continuous work and close support of our development and infrastructure teams are at the core of what makes our Bug bounty project successful, and the same happens with any other security issues we identify and work on. With new vulnerabilities in servers, frameworks, libraries, etc. being identified every day, a security team will always be outnumbered by the sheer number of malicious actors that are present, day and night, on the Internet. It is great to feel we are not alone in this battle; online and at Blip, we are surrounded by incredibly talented professionals that are right there by our side.

If you’d like to know more details about our bug bounty journey, check out this blog post

Behind every button, there’s at least one design meeting happening. Behind every icon, there are a lot of pixels pushed. Behind a neat line of code, there’s a messy Figma file.

I’ll be there for you

A UI Designer is never alone. There’s always a UX by our side that makes sure that our product is not only visually pleasant, but also easy and intuitive, and we work together, every day, for the absolute best experience. In a company full of engineers, product managers, etc, all working towards the same goal, it’s easy to fall into the rabbit hole of what we want the product to be, because year after year, we grow attached to it. That’s where the designers come in, to advocate for the user and make sure we’re building something that not only we value, but that our group of customers will choose above every other product on the market. Together with the UX team based in London, we run user testings, where we give our designs to real people before they ever get implemented, and we see how they perceive them, how they use them, and how we can improve. Doing this at this early stage prevents us from wasting resources and implementing something that we then find out doesn’t work.

Where the wild things are

Imagine your favourite app, picture it in your mind, the buttons, the background, the text… It’s probably a bit hard to know exactly the shape, size, and exact colour of that app’s main button, but I can guarantee you that your subconscious knows it perfectly, and if it changed tomorrow, you would know something was off, even if you couldn’t point out exactly what. This is also true for all the other visual aspects of the app, like background, font size, colours, etc. For this reason, we keep a living thing in our Figma files, it’s composed of atoms, molecules, organisms… We feed it every week, and if we don’t clean it up often, it just doesn’t work. It’s our Design Library, that goes together with our Design System. A Library that is composed of all the components we use in our designs. Every button, every text size, every colour, every icon, every field, hundreds if not thousands of small parts that in the end make our product. It allows us to keep a consistent language throughout every page and interaction, and whenever we want to change or add something, we consider it very carefully.

Come Together

As you know, we take design very seriously, so alongside serious design, we also like to be very serious about our time together as a team. Very serious margaritas, very serious craft beers, and also some very serious dinners happen in our team. We rely on each other a lot, for feedback and all sorts of help, so all of this is important because, at the end of the day, teamwork makes dream work.

Across the world, around 200 countries celebrate Safe Internet Day, dedicated to making the online space a better place for children and young adults. We can proudly say that this European initiative lived to see its 19th edition and bring organizations, private companies, policymakers, NGO´s, parents, teachers, and kids together to establish what should be done to keep them safe online.

It´s no news that the digital environment conquered the young population who now rely so much on mobiles phones, tablets or computers to play, socialize and learn. While according to the Safe Internet forum, younger children tended to use digital technology less often than older respondents, each of them is exposed to similar threats lurking online.

Dangers to look out for:

• Inappropriate content: online search or streaming pages can lead children to stumble upon inappropriate content, be that either violent or adult content. Games present equal challenges where the age ratings will be a good indicator of whether a game is suitable or not.
• Online scams​: children are more vulnerable to common scams like claiming winners or offering promotions. Unsupervised they would more likely fall for these attempts.
• Downloading malware: downloading links from messages, email or simply apps and games extensions can introduce malicious software on a device. These can steal personal information and do other harmful actions on a device.
• Loss of money: when payment details are linked to devices, children can easily or accidentally make purchases online especially for upgrading their online games through in-app purchases. This problem can simply be solved by putting in place some parental controls supported by apps.
• Cyberbullying: means placing an offensive or hurtful public message, image or statement on a social network site or another public forum where that content can be viewed and/or repeated by other people. This behaviour can have a severe impact on children’s confidence, self-esteem and mental health.
• Cyber predators: Using chat rooms, multi-player game setup, or social media opens opportunities for cyber predators to find young children and engage with them while pretending to be their age. The major challenge is getting kids to understand how people can pretend to be anyone they want in the virtual world and be a different person in real life.
• Posting private information: Unwanted information posted online can severely damage a child´s reputation for the long term. Often teenagers would engage in more intimate conversations or send private pictures to someone they like, without thinking about repercussions. Once shared these messages and pictures can be posted throughout social media, causing embarrassment, discrimination shame that can impact the future and self-confidence of a young person for a long time.

How to protect your kids online?

• The earlier you start these talks the better. Foster open communication and explain why they can’t access something instead of just blocking it. This will help them make the right decision in the future. 
• Create together a list of rules and expectations, they will follow them better if they are part of the process. ​
• Define the times when they can go online and for how long they can stay.​
• Talk about the types of websites and/or games they can access and why they are or are not appropriate.​
• What can be purchased online and by whom, to include in-game purchases.​
• Define who they should report problems to, such as strange pop-ups, scary websites, or if someone online is being creepy or a bully.​
• Underline that they should treat others online as they would want to be treated themselves.​
• Explain how people online may not be who they claim to be, and not all information is accurate or truthful.​
• Speak about what information they can share and with whom. Children often do not realize what they post is permanent and public, or that their friends may share their secrets with the world.

What responsibility do we have?

When it comes to our family’s safety, there is hardly room for negotiations. We´ll do the best we can to protect the ones we love, especially if they are vulnerable, like children or teenagers.

Everybody has a role to play in achieving the mission to create a safer internet for the young ones. Organizations can take their share by creating positive online content for kids that is age-appropriate and covering diversity topics. Organizations such as the Safe internet Center or Inhope are there to support the community with means of educating and reporting. As for us, we can contribute by learning more about these threats, sharing our knowledge with other adults in our community and patiently supporting both children around us.

For more information about safe internet day, you can access the Portuguese Chapter.

Do you feel lost and aimless sometimes? This is something that happens more commonly than you think with other people too. So you are not alone in this and it can be fixed. When we talk about the course of our professional career this feeling can have impacting effects, but, if we realize what might be missing and work it out with time and patience, everything lines up on the right track.

So why is a development plan so important? Keep reading the article to find out!

Personal and professional goals

The truth is that the market is more demanding and not everyone is prepared to overcome the rocks that lay in their paths. So, first of all, let’s reflect on what we really want and what matters to us as individuals and as professionals.

We can be led sometimes down to paths that we didn’t expect because they were easier and more straightforward. But, stop for one second and think “is this really what I want for me?” and even more importantly “do I need this for myself to achieve what I want?”

Setting goals will allow you to, on one hand, think about the various possibilities and, finally, select those that best fit your career. This first analysis will help you think outside the box and realize many opportunities that would not otherwise have been considered.

But remember, establishing a development goals plan doesn’t just have to be done only if you’re going to change your role, or upgrade your position, or move to another city. It can happen in any circumstance as soon as you feel the need to grow in your career.

So let’s get started!

The secret is very simple. To be successful in your job you need to follow a strategy of actions and think about the possibilities you can create for yourself. To get there, we have to go beyond the basics, create new growth opportunities, and of course, have a little courage!

That’s why we should start with a self-assessment in which we identify our competencies and weaknesses, evaluate our commitment, and how far we are willing to go to take on new challenges and greater responsibilities.

This is precisely where personal development comes in: self-knowledge. Without it, it may take longer to move on to professional development, but you will still get there!

Personal and professional development

Now you might be thinking “But what’s the difference?“. The truth is that the two live intertwined. Professional development is about our ability to do our jobs efficiently and to improve our performance. To do so, we need to acquire technical and behavioral skills in order to continue our career development. On the other hand, personal development involves knowing our virtues, strengths, and limitations in order to achieve our best selves. These goals, even if they are not career-related, can also help in this matter.

Action plan

Goals and objectives provide a visual map of what your future can look like. They offer a glimpse of what you can achieve if you put your mind to it! This makes it much easier to know what to do next, where to go, and what decisions to make.

We have listed some examples to serve as a basis for your strategic plan. Goals should be seen as something long-term that needs time and dedication. With an effective plan in place, it is much easier to measure the results and understand what decisions need to be made.

Examples of personal development goals:

• Read more books
• Adopt new habits
• Practice a different hobby
• Travel to a new country
• Improve your diet
• Do more exercises
• Learn a new instrument
• Undertake a training course
• Examples of professional goals:
• Learn a new language
• Take training for a particular position
• Develop leadership skills
• Learn about entrepreneurship
• Learn about investments and finance
• Go on an exchange or specialization
• Receive feedback to know what to improve
• Learn more about the company
• Learn how to manage your time
• Work on emotional intelligence
• Improve relationships at work
• To have an international experience

These are just a few examples of the immense opportunities that you can create for yourself! What has worked for a colleague of yours and improved his performance doesn’t necessarily have to work for you as well. That’s why self-knowledge is important.

Furthermore, when these goals are aligned with those of the company they help create a family-like atmosphere where everyone works together and understands their role.

Determination and focus

Now is the time to set your goals to achieve your dreams in your professional or personal life. To do this, you just have to make decisions that are aligned with a purpose. This process requires patience, dedication, and persistence. But we promise that the journey will be rewarding and liberating.

Shall we begin?

In this fast-paced world, we all are focused on improving our performance to its best. And due to the emerging remote and hybrid work models, the search for ways to really deliver value to your company and fight procrastination is even more crucial. We find ourselves in the middle of a vast range of platforms to work in now and that it can be hard for anyone to focus on doing the right job.

If you fit into this scenario and face some obstacles with your productivity and in managing your time, we have listed 10 tips to help you improve your performance. Read on and start putting them into practice! 

1. Don’t multitask

Single-tasking is easier and increases your focus.
While at work, you might tend to spend too much time doing a lot of things at the same time – running from tab to tab in your browser, to your smartphone, and then your company’s chat. However, this kills your productivity. You need to find your way to good management skills… Notifications aren’t ruling your day – your tasks and responsibilities are.

2. Set clear professional and personal goals

It’s always easy to set challenging goals. But don’t they seem out of reach sometimes? Set goals and milestones that are realistic, achievable, and that matter to your day-to-day life, both professional and personal. Forget about incredible milestones that will show a lot of results to your boss and family. Focus on what you can do and work the best to motivate yourself to achieve 100% of each goal.

3. Understand what works for you and what doesn’t

Performing a SWOT analysis on yourself can be a good start to assure your productivity levels. Assess which skills you have that already make you kill at your job and what are those that might need some more improvement. You might find strengths that you didn’t remember and find ways to tackle and overpass the issues you’re facing with productivity. This way, you can understand which areas you should focus on first to deliver work done faster. Also, keep track of your job performance and document your successes.

4. Do more from time to time

Volunteer yourself to do stuff others don’t want to. Even if the project is really intimidating at first, work with your team members – you will have sharp skills and develop new ones in potential new areas of interest that will give you an instant boost of motivation. When facing new projects and discovering new areas of expertise, you’ll gain motivation to excel in the ones you’re already responsible for.

5. Practice team communication

We all know communication maximizes performance at work. However, many times people struggle to speak, asking for clarification when they don’t understand something. This will tend to make your work harder.

Successfully performing people are naturally talkative and look for innovative problem-solving by speaking with their team. Start a culture of speaking with your team and even your management team. Don’t be afraid to talk when you need to clarify all the aspects you might need to do your job perfectly.

6. Plan, schedule, and take breaks!

One great way to improve performance at work is scheduling what works for you in realistic periods to do it. If you have the flexibility to decide when you complete tasks and schedule meetings, this can help you a lot. Start by prioritizing important tasks. Within your schedule, remember to set tasks by priority, making sure there is some time left for breaks or chilling activities. Then fill in the rest of your schedule to deliver minor tasks you might face, or those little helps you should give to your colleagues.

7. Set a time to check email

The inbox can be a source of lost productivity in the workplace, and this tends to get worse as we are now more digital than ever. We all find it tempting to check in regularly and even read messages as soon as they arrive. So, set limited times throughout the day to check email, periods of 5 to 10 minutes are more than enough.

8. Ineffective ways of working go out

If a process you find during your day makes you slow down your pace, start by exploring how to solve this issue by requiring your management team. Continually learn how to innovate and improve all the processes that compose your daily job. Besides making you an expert on it, it will make your work effort smooth.

9. Commit to learning

The best way to improve work performance is by committing yourself to your own professional development. Identify the main gaps within your company and start learning about how to solve them. This can help you be relevant in an innovation process within the organization – and we can tell you, it’s best to keep you interested and focused at your job.

10. Stay healthy a foster work-life balance

If you want to improve your work performance, you must rest, live and breathe. Having hobbies outside your career, sleeping, spending time with family, if it’s what makes you happy, will motivate your general life and impact your work itself.

Free time is crucial for any employee to stay committed to the company, assuring the right levels of productivity and delivered work.

Even the best worker struggles with his/her productivity and with the rush of work forget to put some of these actions in place. Once you identify the flaws, you will be able to adopt techniques that will certainly improve your performance and the whole team.

Realizing where you want to go is the first step to follow the best path.

Do you want to improve your performance but don’t know where to start? The willpower to do better is something natural and instinctive. But we don’t always have the right resources on our side to take the correct direction. Even worse, sometimes we don’t even know what they are and miss great opportunities to be better at what we do.

But keep in mind that effectiveness only nurtures effect when everyone is aligned. So feel free to share this article with your friends..  

Discover our tips that can make all the difference for you and your team!

Make new habits until it becomes natural

There are two different ways to be a better professional from a behavioral and technical point of view. We don’t expect you to follow everything to the letter. The idea is that you find the tools that work best for you and trigger new ideas and perspectives for your future.

Let’s start with the fundamentals

Initiative and adaptability are the main allies of progress.

The best professionals are always looking for opportunities to be better and aren’t afraid to change organizational processes and practices. To begin with, you should focus on resources that involve skills and behavioral changes to make your job more efficient and organized. It is very easy to improve your performance through simple routine actions every day.  Follow along to find out!

Time management

Protect your time! Set your priorities and delegate tasks more effectively. Distractions are everywhere and before you know it you’ve lost half an hour of an important task. So be aware of these escapes and focus on the essentials. Good organization can help you save time, avoid miscommunication, and improve overall efficiency.

Good work habits

Identify the bad and the good habits in your routine and start your day with some alone time before you jump into work. Get to work earlier, plan the day and set a time for your tasks. It is important to also live in the present without distractions, and take essential breaks. 

Apply a system that suits the way you work. You can start with the most urgent tasks and finish with the more operational ones, such as organizing your virtual space and all your documents. These are some habits that will help you be more productive and creative with less effort. 

Leadership

Sometimes the work environment is subject to misunderstandings, so you must be able to observe carefully and respectfully face any decision making. Empathy and interpersonal skills are key to motivating, organizing, and delegating tasks. Good leadership is built on trust and is reflected in motivating others and helping them achieve a common goal.

Communication

Communicating regularly is critical not only to maintain a good working environment but to clarify issues and tasks. You’ve heard this a dozen times, but this is a more common flaw than you realize and it can impact you quite significantly. It’s not enough to do it frequently but to know how to do it clearly and attentively to others’ personalities. 

Actively listening to others is just as important as knowing how to communicate clearly and assertively. Silencing good ideas should be one of the things you avoid in the work environment. Remote or not, you have plenty of online tools to make sure you don’t miss a good, productive conversation. 

Self-confidence

How do you want to try new things and pursue your goals if you don’t have confidence in yourself? Work on your self-knowledge and recognize all your capabilities and skills. If you are confident in your decisions it is easier to deal with challenges and more likely that others will believe in you and be motivated by your ideas. 

Adaptability

Mastering this skill can help you make the most of adverse situations that might arise. If you handle change well you will get along more easily with a variety of situations and personalities within the workplace. This ability will adjust over time. To accomplish this you must also be receptive to change, remain calm, and not jump to conclusions that anticipate a different reality than the one you would experience. 

Initiative

If you seek initiative at work, you shouldn’t be stuck in a comfort zone. You don’t need to be a leader to take the initiative. If you have the vision, empathy, and determination, you will be heard and respected. 

This will help bring more confidence to your work and promote improvements that otherwise would never go forward. Not everything you propose will be the best, it’s important to recognize that, but that’s okay. Stay proactive and prove your worth.

Feedback

Invest in a feedback culture. This will encourage both you and your colleagues to openly share your opinions and insights in a constructive way to boost productivity and quality of work. 

It’s important that you do it regularly, respectfully, and acknowledge everyone’s accomplishments and improvements. If you do so, you are allowing them to do the same with you.

Work set up

This is a topic that sometimes gets overlooked. Uncomfortable spaces can lead to discouragement and lack of concentration. Time optimization and good communication with the team are pointless if you don’t do it in an organized space.

Choose the right colors and decorations, and organize your materials in a way that doesn’t affect your concentration. Work on functional furniture that allows you to access everything.

A more efficient and capable journey

Having the right tools is also essential to do your job well and increase productivity.

Here the opportunities are widely expanded, so we selected some tools that you can easily apply in your day-to-day life.

StayFocusd

Increase your productivity by limiting the amount of time you can spend on websites that waste your time. StayFocusd is an extension for Google Chrome that helps you stay focused at work. Once you reach the allotted time, the websites will be inaccessible for the rest of the day. 

Trello

Trello is a visual work management tool that helps teams brainstorm, plan, manage and streamline their work together in a collaborative, productive, and organized way.

Break Timer

BreakTimer allows you to set up breaks (frequency and duration) and will remind you to take them.

IFTTT

“If this, then that.” is an automation tool that lets you program actions together across applications and services, without the need for coding. By mixing and matching triggers and actions, you can create automation that helps you achieve your goals, and be more efficient regardless of the source/app.

Rescue Time

RescueTime is a tool for tracking your personal productivity. It automatically tracks time, measuring how much time you spend on various programs and websites, without you having to click on anything. 

F.lux

This is a tool that adjusts the color temperature of your display, helping your eyes to rest, and ensuring that you can work on your tasks in quality. Detects the surrounding light and adjusts the screen accordingly to suit your work routine like applying warmer colors if you are working late at night.

Rainy Mood

If you are looking to escape from the distractions and noise of the office this is a tool for you. Essentially designed for sleep or relaxation, this app can be used to calm down and help you concentrate with endless rainy sounds.

How sure are you that you are in the right job? Sometimes the answer is not so obvious, and for various reasons and contexts, we also don’t always make the same judgment as we did 10 years ago or will do in the next 10. But let’s pause for a moment and analyze the present and see what fills us with energy for another day of work and recognize (or not) if this is the right job for us.

To help you out, we’ve compiled a list of the main signs to look out for if you find yourself looking for a job that matches your dreams and needs!

Are you ready to start making “checks”? Let’s do it!

Keep asking yourself the right questions

As time goes by, your life goals may change, so do your aspirations, but there are always essential elements that cannot be overlooked so that at any stage you feel you have what it takes. And yes, it takes courage to claim those “essentials” and try to make sure they are part of your life. But before that, do you think you know how to recognize these positive aspects in your job to realize how important and necessary they are? Sometimes we are under extraordinary conditions, but we don’t recognize them, or on the other hand, we are under basic conditions and don’t demand more.

Maybe you are at the beginning of your career and a lot of doubts and uncertainties crowd your day. Or you already have a lot of experience and have been working in a company for a long time and you want to understand if this is the moment to change or if, in fact, you are in the job you have always wanted, with all the benefits and environments you have aspired to. Don’t worry, it is quite common to not know if you are in the ideal job, especially in the first few months.

Whatever the setting, you must believe that the absolute best is possible. As you continue your journey, you will eventually embark on different moments and challenges and always ask the same question: is this the right job for me?

Let’s find out:

• Pure, positive adrenaline. When your job is super interesting and challenging, you hardly ever stay settled and are always looking to learn new things.
• Your colleagues are more than just work partners. When you work with a team of people you can call friends and confide in a very supportive, empathetic, and genuine environment. You know you can trust them, and they can trust you.
• Time rushes by. When you don’t peek at the time every 10 minutes, that’s a good sign! When this happens, not because you have a too-busy day, but because everything you do is exciting, and you don’t even realize that time is flying by.
• You always manage to find time for yourself.  When you have the flexibility to unplug and assure a proper work-life balance. This is one of the most important attributes of the workplace. You know when to prioritize things but always manage to find time for relaxed moments.
• Is it Monday already? When you start the first day of the week with lots of energy, good mood, and excited to catch up with your colleagues and take on new challenges.
• Weekends are peaceful. When you can recharge your energy without once thinking about work, really switch off, and enjoy quality time with friends and family.
• Bring it on! When you can embrace new challenges and do it with determination and confidence that you will accomplish them effectively, skillfully, and above all, knowing that you have a team that encourages that growth.
• Fair salary. When your salary clearly reflects your responsibilities and the benefits are equal to colleagues with the same functions. Also, you know that the company evaluates it periodically to promote your performance and results.
• You have goals and aspirations.  When there is always room for personal or professional growth, and you keep pushing yourself to give your best because you are motivated to do so, and you know your worth.
• Your role is clear and valued. When you can imagine what your work routine and main responsibilities will be like, your productivity is unaffected and you regularly witness the company appreciating them, in various ways. You know that your work has a purpose and is meaningful.
• Growth with honesty. When the company’s values are aligned with your personal values and the company’s culture and environment continuously motivate you. You know that you will always find an opportunity to talk honestly with your superiors and that your insights will always be valued.
• Your Intuition says YES. Sometimes it is inexplicable, but it can simply happen that your intuition tells you so, regardless of various contexts and factors. You still feel that it is the right job.

A Happy Job: it’s possible and achievable

You also need to understand that it is crucial to realize that self-knowledge is very important and plays an important role when evaluating what makes you happiest and what fills your life in a positive and satisfying way.

Now that you’ve evaluated and interacted with all the signs, can you recognize if you’re currently living the job that’s right for you or if a career change is needed?

Today, more than ever, companies are increasingly looking to ensure wellbeing at work. Given the current pandemic context, the challenges are greater and greater, but they certainly haven’t eliminated the opportunities to provide an answer.

• Have you ever noticed a colleague struggling but didn’t know how to handle it in order to help them?  After all, you are likely to meet a colleague who has gone through a period of increased pressure, discomfort, or demotivation that has jeopardized their wellbeing. But we don’t always have the tools, the means, and the context to do it right. 

Do you want to learn how to invest in improvements to promote wellbeing at work?

Keep reading this article to learn some important signs to look out for, tips for practicing wellbeing, and, ultimately, recognizing its importance in the workplace.

The benefits of wellbeing at work

There are many factors that can affect well-being in the workplace. Yes, the salary matters, the growth opportunities are motivating, the benefits are exciting but… what good is all this if we don’t work in a supportive, empathetic, honest, relaxing work environment?

Companies that already apply and focus on optimizing the quality of life of their employees have increased productivity, improved employee retention, and individual and overall results boast significant improvements.

So let’s get to it. Because we spend so much of our day working, we can’t let that sacrifice our physical and mental well-being.

So take a break, grab a fresh drink, and navigate through some of the best resources and suggestions for a more fulfilling life, properly booked and loaded with nothing but positive energy.

Spot the signs

And speaking about positive energy, ironically we need to mention the elephant in the room: the signs that we or someone may be going through a phase of poor wellbeing. 

And when this happens we can risk negatively impacting performance and efficiency as well, which in return will lead to more stress, pressure, anxiety, and sometimes depression.

But, first of all, keep in mind that everyone is different and we all react in different ways to some triggers, so don’t immediately assume something without first taking the time to get to know your colleague well, and spot changes in their behavior that should be valued.

Here are some signs you should be aware of, some more obvious than others:

• Poor concentration
• Being easily distracted
• Weeping
• Talking less and avoiding social activities
• Talking more or talking too fast, jumping between topics and ideas
• Finding it difficult to control emotions
• Drinking more
• Long-lasting sadness or irritability and short temper
• Aggression
• Worrying more
• Finding it hard to make decisions
• Feeling less interested in the day to day activities
• Low mood
• Feeling overwhelmed by things
• Tiredness and lack of energy
• Sleeping more or less

When you notice some of these signs in a colleague, it is very important that you get a good reading of their behaviors and know how to start a conversation with them. Questions like “How are you?”, “What’s been happening for you lately?”, “How is your family?”, can help start a conversation and attempt to help understand what that person is facing. And you could follow up with something like this: “Is there anything more or different that I can do to support?”

And, if by chance, you don’t have that much trust and a solid relationship with that person, start precisely by addressing that point honestly, acknowledge it, and explain that despite that, you’d like to know how he/she is doing.

Best tips to practice wellbeing

It is no wonder that the fastest-growing companies are those that care about their employees. As you will see, it is extremely important to consider well-being at work very important, for you, your colleagues and the company’s results. 

As a company:

• Create a culture of mutual respect
• Promote improvements in the workspace
• Promote collaborative activities
• Encourage physical activity
• Encourage openness
• Provide moments of rest and relaxation
• Change some habits to help improve the mental state
• Discourage people from working long hours  

As an individual:

• Take time out for things you enjoy
• Be active and eat well
• Nurture relationships and connect with others.
• Get involved and join in
• Be as much social as you can
• Build your confidence
• Practice new hobbies
• Set realistic goals and deal with tasks one at a time
• Reach out for help when you need it
• Get enough sleep and rest

Everyone wins when the environment is collaborative, comfortable, and most of all, genuine.

But keep in mind that it is ok not to have all the answers and not know how to proceed in every situation. As long as you can carefully listen to someone, or acknowledge your struggles and know how-to guide in the right direction, you’ll be well on your way to contributing to a workplace where everyone’s wellbeing is prioritized. 

Everyone talks about sustainability and environmental awareness, but the truth is that we spend a good deal of our time in the office not realizing what improvements we can make to make it more eco-friendly. And most important, the impact it can have on a more sustainable future.

Did you know that green offices provide a more caring, conscious, and happier environment, unlike conventional ones? If done correctly and persistently it can encourage employee proactivity, a sense of belonging, and promote a more lively and joyful space. It might seem obvious, but unfortunately, that’s not the case. Human behavior is the biggest obstacle when we talk about going green. Changing habits can present some resistance, but when carried out collectively, they can have a tremendous positive impact.

First of all, we should be aware of the problem, and secondly do what we are doing right now: look for solutions, improve habits, and ultimately go green.

Are you ready? Follow our recommendations below and start fresh.

Despite all the efforts, campaigns, and articles (such as this one), people continue to conduct consumer behavior that privileges the use of plastic, are somewhat oblivious to global changes, and unaware of the responsibility of protecting the environment and wildlife.

As you can see, the reasons to consider these changes outweigh any hesitation you may have. It might be true that your position may not allow you to have a say when it comes to the lighting in your office, but that doesn’t mean that you can’t adopt behaviors for the benefit of an environmentally friendly one.

So what can we do?

To make it easier to protect the environment but also to promote a healthier and more environmentally friendly lifestyle, we have collected the following tips:

1 – Unplug all the devices you are not using for instant energy savings.

2 – Use microfiber cloths instead of paper towels.

3 – Use as much natural light as space can offer.

4–  Establish a sustainability initiative with a green team.

5 – Invest in smart power strips. It will help you waste less power.

6 – Fill the office with plants. A great choice not only for decoration but also to help reduce the office’s carbon emissions and purify the air.

7 – Donate or recycle old electronics.

8 – Install LED bulbs as they use around 75% less energy than standard light bulbs.

9 – Switch to glass storage containers instead of plastic.

10 – Compost your food waste and start a green garden.

11 – Go paperless, avoiding printing emails and having paperless meetings.

And if you can’t reduce, then reuse.

12 – Have you heard of carpooling? Take this opportunity to engage with your colleagues.

13 – Lower your environmental footprint by using your bike to go to the office a few times a week.

14 – Create a recycling center and place one box near the printers.

15 – Get a coffee maker that doesn’t need paper filters and then compost the coffee grounds.

16 – Encourage the use of environmentally friendly cleaning products.

17 – Replace plastic cutleries with reusable ones.

18 – Encourage your company to install electric hand dryers instead of paper towels.

19 – Be rational about your lunches and don’t throw away leftovers. There are many ways to turn it into another delicious meal!

20 – Support and organize sustainability challenges so everyone gets involved.

It is a team effort

Are you as excited as we are to start making these changes? Chances are that your company already has a series of rules and actions in place that address these concerns.

Think about it, you could be responsible for reducing tons of waste just by starting to live a more eco-friendly lifestyle.

Work-life balance is an increasingly current and discussed topic further heightened by the pandemic global context. But do we really understand the true meaning of this concept? 

First of all, you should ask yourself this question: can I successfully and comfortably conciliate my personal commitments and responsibilities with my professional commitments and responsibilities? 

If you can’t get a clear and immediate answer, then this article is for you! 

Find out the best strategies to properly distribute your time to satisfy both your personal and professional interests.

Where should we begin?

Above all, you must realize that balancing time doesn’t mean distributing your time equally. But understanding the best conciliation according to importance and well-being.

Today, work-life balance is one of the most important attributes of the workplace, often ahead of financial compensation.  

It shouldn’t even need to be mentioned at this point but it is true: workers who feel they have a better work-life balance tend to work harder than workers who feel overworked. And when we say “harder” we are talking about performance, creativity, and proactivity to develop solutions that otherwise would never exist.

On top of that, the chances of feeling that they belong to a group and the sense of accomplishment and purpose suppress anything else.

Companies that promote work-life balance have more satisfied employees and the investment in time flexibility measures can solve problems such as health risks, decreased productivity, employee turnovers, absenteeism, mental health, stress, and overall life satisfaction. 

When we talk about work-life balance we cannot ignore two major things: motivation and autonomy. This means autonomy over your time and sense of purpose in working on something meaningful that really matters on your own terms. When this autonomy and motivation are in alignment we are stimulating increased productivity, engagement, and satisfaction.

6 strategies to easily improve your work-life balance 

Do you have control over your life or does it seem like you do?

Well, it’s something that can really be worked on and improved together with your company. But if your company applies flexible measures and you are not aware of the improvements you can make in your habits and perception of what really matters, what’s the point?

Here is how you can make a start and enhance your work-life balance: 

• Structure your week

Take a look at your diary at least one week in advance and find gaps and opportunities to prioritize your well-being first. Review your tasks carefully, there is always something that can be improved to allocate more time to the tasks that really matter. The more control you have over your work, the less stressed you will get. Even more important, ask for help whenever necessary and make time for yourself every day. Be it 30 minutes or 1 hour, It will help not only your relationships but your career as well.

• Support a flexible work culture

First of all, it is important to understand that each of us works at different times of the day, especially within a global company. So we can find ways to work together without the “pressure” of response times. You can always find and suggest practical alternatives like putting a note in your signature to let your colleagues know in advance. You are not alone in this, chances are they are facing the same struggles, so support and stimulate a stronger support system.

•  Turn off notifications and establish boundaries

Digital communications have come to make our time easier and more flexible, but they have also come to steal the time we need for our personal commitments and needs. It is very important to create boundaries to maintain healthy digital habits. And this should be communicated clearly with your supervisors and colleagues as well, so they know you won’t be responding and paying attention after hours. Turn off notifications from apps like Slack or your email outside of work hours and set limits on accessing them in a personal context. Before you know it, you have already wasted an hour of your family evening hooked on to the digital.

• Create daily to do lists

Sometimes it can be hard to disconnect from work especially when you just had a very intense day with many meetings and tasks. This can cause anxiety and stress in order to catch up on everything the next day. A good way to feel more confident and prepared is to organize and write down your thoughts, ideas, and follow-up tasks at the moment. This way you will save up time the next day and be able to properly leave work at work and enjoy the present. 

• Don’t be afraid to unplug

Using time more efficiently also means knowing when to stop and take a break. This should not only be a habit you make in your daily life but something you actually schedule. It should be encouraged by companies and colleagues! Regular breaks will allow you to recharge and will give you the energy you need to perform your tasks successfully. 

• Take some time off

Vacation time has finally arrived and you feel that this will be the time to recharge all energies and really switch off. It sounds both obvious and easy, but it’s not. And it shouldn’t be the only time to do it. As you noticed in the previous tips, there are many ways to do this during work time. It needs to be clear to your team that you will be completely absent but, above all, it should be clear to you as well. Don’t peek at your email, don’t wake up to see if you have any notifications. Relax, recharge, and refocus

In short

Before you can improve things, you have to realize the problems. 

Over time you will easily gain control over your life and make the right choices and habits for your personal well-being. It’s not just up to your company, it’s up to you. Before instructions are the will and the motivation, and you have to want it.