Whether you’re watching a football game on TV or in the stadium, there’s one phrase every fan, player, and coach hates to hear: “There’s a flag on the play.” Just when you thought your team broke off a massive run, the officials come in to spoil everything. But, even if you’re a life long fan, you might now know why the referees toss a yellow flag to signal a penalty.

The story actually begins in 1941, in much more humble surroundings than today’s NFL stadiums.

The early days of football

While baseball may be known as America’s pastime, football also has a long history. The sport evolved from rugby and quickly took hold; Walter Camp developed the rules of “American football” in 1880, and the NCAA itself was actually founded in 1901 to help regulate the collegiate game.

Warner defined things that we all recognize, like the line of scrimmage and legal blocking, but the game was still a far cry from modern football. Players wore little padding; over time, leather helmets were introduced. The field of play was also different, with the goalposts standing on the goal line instead of at the back of the endzone.

Officials also used a different technique to signal penalties.

Throwing the first flag

During the early days of football, referees would blow a whistle or a horn to signal a penalty. That system, however, had a fatal flaw: a lack of clarity. Since whistles also marked the end of play, it wasn’t always clear what the officials were signaling. Fans would also bring their own horns and noisemakers, adding to the confusion.

In 1941, Youngstown College head coach Dwight “Dike” Beede decided that something needed to change. Since football games were such noisy affairs, he wanted a visual signal that a penalty had been committed. “I came home from practice and asked my wife to do a little sewing,” he remembered.

Beede’s wife, Irma, cut up an old Halloween costume and stitched the fabric together with some bedsheets. She created four flags, each measuring 16 inches square, and anchored by drapery weights. The flags were tested in Youngstown’s next game; referee George McPhee found the new signal to be effective and kept using it.

The penalty flag grows up

In 1943, McPhee joined the Western Conference, the predecessor to the modern Big 10, and brought his penalty flag with him. When he used it in a game between Ohio State and the University of Iowa, however, someone would take notice.

Western Conference commissioner John Griffith was ina the stands that day and was impressed by McPhee’s penalty signal. He decided that, starting the next weekend, all Western Conference officials would throw flags whenever a player committed a penalty.

The NFL adopted the penalty flag in 1948, but the fabric was originally white. The league shifted to yellow in 1965, coinciding with the rise of color televisions. While flags are no longer filled with ball bearings after Orlando Brown was infamously hit in the eye, they’re still more or less the same technology that was invented years ago.

While Beede changed the face of football forever, he was never looking to make history. “He told me he saw a problem and offered a solution,” Ken Breyer, a former student manager and sports information director at Youngstown State, explained. “He had no idea how big it would grow.”