In non-video games, there are usually physical components that restrict the actions that players may take. These tangible objects are embodiments of the rules that the game enforces. For example, in Monopoly (Darrow 1934), the design of the board, with its fixed spaces is a representation of rules that are enforced by the game. You cant break the rule of their being 9 spaces per edge (not counting corner spaces) without changing the actual physical board. We call the rules that are enforced by the game, the gameworld rules.
This distinction is important, because it helps understand the case of videogames. Practically by definition, a videogame has no tangible objects. (those that do exist, are usually described as tools that allow the player to communicate with the game, and vice versa, see Interface in this ontology) So, if gameworld rules are those that are enforced by the game, what does that mean in the case of digital games?
Computer games are highly spatial (Murray 1997). Most games convey a notion of place to the player. His or her participation in the game is within the boundaries of this virtual world. This world, not necessarily the same as the physical world we inhabit, is still subject to its own constraints and has a consistency that allows the player to recognize it as a world. These constraints, or these characteristics that make it identifiable as such are what we call the gameworld rules. The most obvious gameworld rules are those that attempt to emulate real-world rules. For example, physics. These rules may constrain the way that the game characters can move within the game, or maybe allow them actions that are not generally possible (just as falling without taking damage)
Strong Example Most platformer games have gameworlds in which there is a simulation of gravity. The player controlled character will fall (usually towards the bottom of the screen) when there is nothing underneath it to hold it up. In this case, the existence of gravity as well as the particular things to which it applies is a type of gameworld rule.
Strong Example In Elder Scrolls: Oblivion, there are not many gameplay rules enforced on the player. Instead, the player relies on the game-world to provide a structure in which to play. It is a simulator based rule system which relies on the extension of not only gravity, but time as well. For instance, there is the notion of day and night. If a player travels from one village to another, they may leave during the day and arrive at night (depending on how far the journey). At night you may not be able to enter a person's residence or do certain tasks because the character's may be asleep or shops may be closed. This in turn encourages the player to do other tasks that emulate real world rules, such as sleep.
Darrow, C. (1934). Monopoly, Parker Brothers.
Murray, J. H. (1997). Hamlet on the Holodeck: The Future of Narrative in Cyberspace. New York, The Free Press.