Need a computer vision expert for a virtual reality project (But will prefer Malik Tutor or a professional with lower budget )

Abstract

This work presents the design and implementation of an embedded augmented reality game, called MarkerMatch. Augmented reality is a technology that directly contributes to the game interaction experience by enhancing user’s sense of immersion. Current research in embedded augmented reality enables the creation of dedicated hardware capable of executing augmented reality applications. This favors the insertion of augmented reality capabilities in small electronic devices, such as cell phones, handhelds, head-mounted displays and even the development of new ones. The ARCam framework was used for game development, since it provides project designers with all the basic infrastructure needed by the game. Some user tests show that the tested subjects enjoyed the game experience and it proves a point: it is possible to create an augmented reality game completely in hardware with no software involved.

1. Introduction

Augmented Reality (AR) makes use of computer vision algorithms in order to superimpose virtual information 2D or 3D, textual or pictorial - onto real world scenes in real time, enhancing user’s perception of and interaction with the environment [4]. Nowadays, augmented reality is applied in different fields, such as entertainment [23], medicine [5], manufacturing and repair [4], and training [19]. The technical challenges lie in determining, in real time, what should be shown where and how.

Traditionally, augmented reality systems place virtual objects in the real world using fiducial markers. Such artificial markers are used to support camera position and orientation tracking by the system, and are intrusive to the environment. Figure 1 illustrates the use of such fiducial

Figure 1. Marker based augmented reality example

markers in order to place a virtual statue on the real table.

The concept of augmented reality is directly related to augmenting users’ perception, specifically the users’ vision. Therefore users need to wear HMDs or similar devices in order to obtain the information enhancement previously mentioned. More important than that, many augmented reality applications are made to provide support to users in their daily and common activities. Therefore, there has been an expanding tendency to seamlessly integrate daily used equipments into common platforms with support to mobility. Continuous advances in device miniaturization, allied with the emergence of various wireless communication technologies, universal plug-and-play devices and powerful portable processing units has opened the door for research on wearable platforms.

It’s natural the evolution of augmented reality desktop platforms into something closer to the user. The terms mobile and wearable must be considered part of such evolution, and for this to happen, the miniaturization and specificity of devices must occur. Embedded augmented reality [22] refers to the research area that aims enabling the mentioned evolution. It researches how augmented reality applications can fit into small devices in a way that users can experience them seamlessly.

An embedded augmented reality application may vary from a software running on a small processor, attached to the user’s clothes, to a dedicated circuitry running on the user’s glasses, for example. The later is an example of the most extreme embedded augmented reality, since it considers the augmented reality application being a chip that can be integrated into existing devices, such as cell phones, PDAs, or can give birth to new ones. The case study developed in this work, called MarkerMatch, exemplifies this class of applications. The entire application, after its prototipation phase, can be packed into a single and dedicated chip and will be further detailed in Section 4.

This paper is organized as follows. Section 2 lists some work related to embedded augmented reality applications and games using augmented reality. Section 3 details the ARCam, which is the framework used as basis of this work, explaining what is provided and how it enables the development of pure embedded augmented reality applications. Section 4 describes the game used as case study, its architecture, implementation and the results obtained. Section 5 focuses on the user tests performed. Section 6 gives final considerations about the project and some directions are given for future work.

2 Related work

The idea of embedding devices with augmented reality capabilities is complex, mainly due to processing power constraints. Another factor that makes it difficult is the vast amount of ways of programming the target device. Apart from augmented reality desktop applications, it is not always possible to use high level programming languages and APIs. In the case of cell phones and PDAs, one must use the APIs provided by the vendor in order to obtain maximum performance. When working with FPGAs (FieldProgrammable Gate Arrays), for example, hardware description languages are used, such as VHDL (Very High Speed Integrated Circuits Hardware Description Language) [3], and they normally present a higher learning curve.

There are a lot of embedded computer vision systems capable of recognizing input from some type of sensor, and taking actions based on the information obtained [13]. The main difference perceived when comparing computer vision algorithms and augmented reality ones is that the later must complete an entire pipeline, from information acquisition to visualization. Computer vision algorithms are just part of the application. Because of that, it is not so common to find embedded augmented reality examples as embedded computer vision ones. That being said, we will focus our related work section on augmented reality applications developed for mobile devices, and mobile applications that were built over dedicated architectures.

The Wikitude World Browser application [21] presents data about users’ surroundings, nearby landmarks, and other points of interest by overlaying information on the real-time camera view of a smart-phone. It currently offers support to a few cell phone models (G1, G2 and myTouch), due to the processing power demanded. The developers of Wikitude also made available an API that allows the development of markerless augmented reality applications, providing developers with the tools to either create their own Android [10] augmented reality applications or enhance their existing Android applications with an augmented reality camera-view engine.

Similar to the Wikitude, there is the Layar Mobile Augmented Reality Browser [18]. Layar is a free mobile phone application which shows what is around the user by displaying real-time digital information on top of reality through the camera of the cell phone. It’s available to more phone models than the previous Browser, such as iPhone 3GS, T-Mobile G1, HTC Magic and other Android phones, and comes pre-installed in some specific models.

The Augmented Reality Tower Defense [7] was developed for Symbian OS, and is a game that uses markers to get the position of the phone camera. The cell phone is used both as display and 3D mouse.

Another example of game that uses a mobile platform is the ARhrrrr [14]. It is an augmented reality shooter for mobile camera-phones. The phone provides a window into a 3D town overrun with zombies. Pointing the camera at the game map mixes virtual and real world content. Since it runs in a Tegra-based device and there is no comercial device that can run the game yet, it’s just a concept demonstration. It takes benefit of the capabilities provided by the tegra chip in order to supply all processing load required by the game.

The AR Drone [16] is a different example. Instead of using the capabilities provided by some existing platform, the authors built a dedicated one. The embedded computer system developed includes an ARM9 RISC processor at 468MHz, 128MB of DDR RAM, wireless capabilities, 2 cameras and other sensors, and runs Linux. The developed platform is capable of detecting markers, calculate their position and orientation and send this information along with visual feedback to an iPhone. It comes with two augmented reality applications pre-installed on the platform.

The case study performed in this work is similar to the AR Drone in a way that both have dedicated platforms for running the augmented reality applications. Some notable differences are that the AR Drone uses a general purpose processor, while the game developed in our case study makes use of a number of different hardware modules, each one running in parallel and performing a task of the game or a stage of the augmented reality pipeline. Another difference is that AR Drone’s applications have been developed in software, while the modules in our case study were completely developed in hardware, therefore there is no software is used.

3 ARCam framework

The MarkerMatch project was built using ARCam’s (Augmented Reality Camera) infrastructure [11], which is an embedded framework for development of augmented reality applications on FPGAs using fudicial markers (similar to the ID-based ones used by ARToolkitPlus [23]). This framework simplifies developers work by providing several hardware modules directly described in register transfer level, that can perform every operation of a markerbased augmented reality pipeline, according to Figure 2. This way, the developer doesn’t need to concern about some infrastructure’s development details, required by every augmented reality application, such as camera capturing, marker detection and image displaying.

Figure 3. ARCam’s architecture

In order to achieve its goal, ARCam was designed to require the minimum amount of hardware’s resources, enabling the developer to succeed in creating its application even on a very restricted environment. It is coded completly in VHDL and no soft-core is used in its operations, therefore it is a pure hardware solution, employing thus all its potential. ARCam’s processing unit is an Altera Stratix II FPGA with 48,352 adaptative look-up tables (ALUTs), 2,544,192 bits of RAM, 36 DSP blocks and 144 multipliers.

The framework provides two sets of components: an algorithm pipeline, including the most used algorithms in augmented reality; and a group of modules to control platform-related components, as image sensor control and video decoder interface.

The implemented augmented reality pipeline algorithms are the following: binarization, gray scale, labeling, mean filter, edge detection, generic convolution, centroid estimation, square detection, 3D object wireframe renderization, marker detection abd pose estimation. The other modules consist of video decoder interface, video encoder interface, image sensor control, real/virtual selector and memory modules. A block diagram of the platform architecture is shown in Figure 3.

4 Case study: the MarkerMatch game

A guessing-game is a game in which the task is to guess some kind of information, such as a word, a phrase, a title, or the location of an object [25]. Many of such games are played cooperatively. In some of them, one of the players knows the answer but cannot tell the others; instead, he/she must help them to guess it correctly. Some examples of famous guessing games include Battleship [20], Charades [9], Hangman [17], Mastermind [15], Pictionary [24] and Guess Who? [12].

The Battleship game was invented by Clifford Von Wickler in the early 1900s, and further published by many companies. One well-known variant of the game was produced by 3M in 1974, in a series of eight abstract strategy and puzzle games called “3M Paper Games for Travel & Leisure”, as illustrated in Figure 4.

Our case study, named MarkerMatch game, has as objective to develop an embedded two-player guessing-game, similar to the famous Battleship game, in which they compete to find the misterious pattern chosen by the game master, which in the case of MarkerMatch is the computer itself.

A prototype of MarkerMatch was built using an image sensor and a monitor, both connected to an FPGA. Some ID-based markers were also used. Figure 5 presents the components.

4.1 Mechanics

MarkerMatch is a turn-based guessing game, played by two people, in which the winner is the one who first reaches the maximum score defined. Its match consists of many rounds. In each round, a marker not yet used is randomly

Figure 2. Augmented reality pipeline

Figure 4. Paper game from 3M (1974)

chosen and the players, who possess a pack of cards (each one containing a marker), will try to guess the chosen marker. The player who correctly guesses it is considered the winner of the round and sums one point.

In case both players fail to guess the marker, a new tip is revealed. The concept of tip in MarkerMatch corresponds to revealing the color and position of a randomly chosen fraction of the correct marker. The tips continue to be given to the players while they do not guess correctly or there are no tips left to show. In the beginning of each round a tip is automatically revealed.

A new round starts every time one ends, unless one of the players has reached ten points, which is the game stop condition that dictates who wins the match. In order to make the game more dynamic, intuitive and fun, augmented reality is used. This way, the players must show the markers to a camera. Using a dedicated circuit, the marker shown is verified to be the right one or not. Then, the information is sent to a monitor that displays the result to the players.

Thirty-six ID-based markers are used, as shown in Figure 6. Each marker is composed of a 6x6 grid of small binary squares (assuming black or white color), which can be translated into a unique ID. The marker’s image captured by the image sensor is analyzed and further associated with its corresponding ID. This way, markers are recognized in MarkerMatch.

The game uses the display attached to the FPGA to provide the players with all necessary information regarding the match. Figure 7 illustrates the game screen.

Region (1) exemplifies the augmented reality used on the game, by adding information to the image captured by the camera. In case the marker is guessed correctly, it is highlighted in green; otherwise, the red color is used. This region can also be used to display other information such as a game-over screen, for example. Region (2) displays the partial score of both players. In addition, it also indicates to whom belongs the turn. Such information is highlighted by the square placed on the right of the player’s name. Region

(3) provides the players with some tips regarding the correct MarkerMatch was implemented and organized based on the infrastructure provided by ARCam. This infrastructure was slightly adapted as required by the game and the game control module was built over it. This way, the system design is organized into subsystems, which are: Video Capture, Video Visualization, AR Processing and Game Control. Figure 8 shows an overview of project’s architecture.

Figure 5. FPGA board, image sensor and ID-

based markers used in MarkerMatch game

marker to be guessed.

4.2

Architecture

Figure 6. Grid related to a specific ID

Video Capture and Visualization subsystems are originally from ARCam, with only very few changes. The Video Capture subsystem is responsible for interfacing both image sensor and FPGA and it also stores the real world image. It was adapted to capture and store 240x240 pixels images, as the left-side bar is used to supply information to the user. The Video Visualization subsystem is responsible for multiplexing real/virtual world images and interfaces the FPGA with the screen, via a VGA output. The original VGA controller was modified to generate different address signals to the many memories that store each region of the final image.

The AR Processing subsystem is also part of ARCam and among the available modules, MarkerMatch required Figure 7. Information displayed on the monitor

the following ones from the augmented reality pipeline: binarization, edge, quad and marker detection. Since the subsystem’s main purpose is to identify markers, it was modified to work with 240x240 pixels images. Another important functionality is to highlight the identified markers on the screen. This is done by masking the pixels of the identified marker using an auxiliary memory. The Visualization subsystem uses the stored mask to properly render the marker on the display.

The Control subsystem functions as the brain of the game, being responsible for coordinating and controlling the other subsystems. The most important modules on the Control are: Game Controller, Tip Generator and Marker Generator. The Game Controller defines the finite state machine of the game, illustrated in Figure 9, which consists of

Figure 8. Project’s architecture

the following states:

· game initialization

· tip generation

· player turns

· end turn/game.

Figure 9. Game state machine

On the initialization states, the Game Controller configures the system as required by the game. Such configuration step involves resetting both player’s scores and tips already shown and requesting a new (not yet used) marker to the Marker Generator along with a tip to the Tip Generator. When all the information is ready, the game begins with player 1’s turn.

A second process, which is executed in parallel to the Game Controller’s state machine, is responsible for verifying if the marker identified by the AR Processing subsystem is the correct one. This process updates the identified marker’s information at every identification procedure execution and makes it available to other procedures.

Players’ turns are alternate, while nobody guesses the correct marker. When someone shows a wrong marker, Game Controller informs AR Processing subsystem, which will fill the marker’s mask with a red color, so that Video Visualization can paint the marker in red on the screen. The same happens when the correct marker is shown, but this time the color will be green. The player can only try to guess the marker once per turn, as the state machine will only repaint the marker’s mask until the other player’s turn begins (and not check for marker identification).

When a wrong marker is recognized, the Controller no longer accepts any attempt to show a new marker on the current turn, so that the player must press the button on the FPGA to pass turns. Even if the player shows another marker, the Controller and the AR Processing subsystems will not take into consideration the ID of the marker and will fill the mask of the marker with the red color, as for the first try. The choice for changing turns by using the push-button was simply a way of guaranteeing that the user acknowledges the end of his/her turn.

By pressing the FPGA button, there are two possible situations: changing the state to the next player or requesting a new tip. It depends on whether it is player 1’s turn or player 2’s, respectively. In case it is player 1’s turn, then the round is not over and player 2 gets a chance to guess. If it is player 2’s turn, then the round is over and nobody correctly guessed the marker. In sequence, the Controller checks if there is any tip left and, if not, it is player 1’s turn again; otherwise, before going to player 1’s turn, the game waits while the Tip Generator provides a new tip.

When the guessed marker is the right one, the Controller checks whether the player of the turn has reached the score that makes him/her the winner. If it has not, a new match is started with a randomly chosen new marker and the tip mask is erased. Otherwise, it has reached “game over” state, in which a picture is shown on the screen indicating the end of the game and the winner. Once in this state, a different FPGA button must be pressed to start a new game.

4.3 Results

In order to analyze the characteristics of MarkerMatch as an embedded augmented reality application, two important factors must be taken into consideration: the processing time, because the game is an augmented reality system and has to give a feedback fast enough to create an immersive user experience; and the amount of hardware used, as the resources are relatively scarce in embedded environments.

Most of the execution time in the game is due to image processing to correctly detect the markers. The game’s controlling tasks, which run in parallel to image processing tasks, are comparatively faster and consume only a few clock cycles, thus it will be ignored in timing analysis. Using a timer in hardware and an FPGA in-system debug tool provided by Altera [1] (SignalTap Logic Analyzer), it was possible to determine how many clock cycles are necessary to execute each module procedure. The clock speed at which the entire system runs is 25 MHz, therefore the execution time can be calculated. Table 1 presents the number of clock cycles and the execution time per module. However, these numbers should not be summed, as the ARCam framework implements these functions as a regular pipeline. Thus, in spite of the first few frames, the execution time of the whole process is approximately the time of the longest pipeline stage, what results in a 87 FPS (frames per second) performance.

Table 1. Clock cycles and execution time per module.

	Clock cycles	Execution time
Binarization	57,600	2.30 ms
Edge Detection	288,000	11.52 ms
Quad Detection	47,023	1.89 ms
Marker Detection	4,408	0.18 ms

The amount of hardware resources taken by the system is also a crucial factor, because this is a very limited environment and the smaller the system is, the faster it can be and easier will be to produce it. As consequence, less power is dissipated. After the compilation process, the compiler tool generates a report with several indicators of resource usage. The most important indicators are the amount of ALUTs, registers, memory bits and DSP (digital signal processing) elements. Table 2 presents these indicators per subsystem. The used resources represent 50% of the available ALUTs, 20% of the available registers, 78% of the available memory and 67% of the available DSP elements.

The developed case study shows the feasibility of implementing an embedded augmented reality solution completely in hardware, in a relatively short period of time. The use of ARCam framework as base facilitated the development process while decreasing the time spent. It took about 45 days to conclude the entire embedded application, including conception of the game, coding and testing. This period also considers the time spent modifying the game with the suggestions gathered from the user tests.

5 User tests

A total of ten users were asked to play the game in order to identify potential problems on the MarkerMatch gameplay. All users were familiar to computer games and augTable 2. Resource usage indicators.

	ALUTs	Registers	Memorybits	DSPelements
Video Capture	195	152	1036800	0
Video Visualization	1193	185	288000	0
AR Processing	21072	8678	529367	178
Game Controller	1580	874	123136	16

mented reality projects, their average age was twenty four ears old and all of them were male. For this case study, the amount of ten users has revealed to be enough for the testing activities due the similarity on all answers.

After introducing them to the elements on the display and to the mechanics as well, the users were observed, two at a time, playing against each other for at least fifteen minutes. During the test, notes were made and opinions were taken from the players. When the game was finished, they were asked some questions which could provide important information about how to balance the game.

When asked about the amount of markers that they had for guessing the right one, the average answer was twenty two, a significant reduction comparing to the initial thirty six proposed. They told it would turn the game into a more active challenge with easier matches. Based on their answers, there was a reduction on the number of rounds needed to finish a match as well. According to them, the ideal number of rounds should be five, exactly half of the ten proposed. This way they could finish the match, because all the matches observed did not go until the end, to the winner’s screen.

Another question made was about the use of timing for each player attempt. Except for two of all users, they told this feature was primal for the gameplay, giving it more excitation and reducing the duration of a match.

As regards to the number of tips given in the beginning of each round as well as those given after the second player attempt, we were told that one tip was appropriate for both situations.

At last, asking for a feedback about when the marker is guessed right or not, all users informed that the green and red squares were enough to provide feedback about the attempt performed.

Figure 10. Semantic differential scale used

Based on these opinions and answers, few tweaking and improvements were made on the game code and mechanics, to adequate the MarkerMatch game to user needs.

5.1 Semantic Differential Scale

Once finished the user tests, where important information about the gameplay were collected for implementation adjustments, a second activity with all users was made. This time, they were asked to evaluate the game according to a semantic differential scale [8] with sixteen subjective parameters considering three different views about the game: (1) the gameplay, (2) the game feel, and, (3) the use of the game as an application.

The gameplay overview comprehends aspects like the system usability, easiness on playing and immersion. As regards to the game feel, we considered aspects related to esthetics or user feelings, and were evaluated parameters most related to fun. Finally, considering the game use, the user interaction, socialization and innovation were aspects analyzed.

All ten users filled up the scale, which can be understood as a discrete visual analog scale [6], according to their first experience on playing the game. We made use of a seven point scale, since odd numbers are more suitable for this kind of scale because it indicates the midpoint, and more than seven points could confuse people on placing their point of view on the scale [2]. The average result is demonstrated in Figure 10.

As it is indicated, the MarkerMatch case study obtained satisfactory results, especially for a first test with a beta version. The average opinion about each parameter was placed between the first and fourth boxes (which represents the average between good and bad) for all those three head groups discussed above.

The worst remarks were made regarding the capacity of the game to surprise or educate the player, which was expected, as MarkerMatch is a guessing game. On the other hand, the best remarks were made for the challenge and user interaction promoted, which is usually the focus of augmented reality projects [5].

6 Conclusion and future work

This article presented MarkerMatch, a case study for the embedded augmented reality framework ARCam, with the main purpose of evaluating the feasibility of creating an augmented reality application completely developed in hardware. In order to measure the success in this task, several criteria were taken into consideration: amount of hardware required, game frame rate and user evaluation. For the good results presented, the application managed to fulfill every aspect properly.

Due to the pipeline-like implementation inherited from ARCam, the total amount of time needed by MarkerMatch to run (a single frame time) is about 11.4 milliseconds, enabling real-time user interaction. After synthesized, the complete game took about 60% of the FPGA area resources. Therefore, there is still room for enhancements in both game features and implementation.

The user evaluation brought forth many issues that, once implemented, richly improved the game, especially by enhancing a critical matter in augmented reality applications, which is the user interaction.

The case study was also useful to determine whether or not the framework could help to implement an application in such a challenging environment. Indeed, it helped, as the development team composed by 3 people finished its work in about 45 days. Although it was not tried to recreate the game without ARCam’s support, it is believed based on our experience that such task would have taken months to be accomplished.

As future work related to the hardware implementation, the process of image binarization is being improved, by considering dynamic thresholds instead of static ones. This way, the application will suffer less from environment light variations. As future work related to the game, embedded augmented reality offers the possibility of using the system in outdoor scenarios, such as treasure hunts, by using embedded augmented reality on mobile phones or wearable devices.

7 Acknowledgements

The authors want to thank MCT and CNPq, for financially supporting this research (process [omitted for the blind review]).

References

[1] Altera. Fpga cpld and asic from altera, Jan. 2010.

[2] D. F. Alwin and J. A. Krosnick. The reliability of survey attitude measurement: The influence of question and respondent attributes. Sociological Methods Research, 20(1):139– 181, August 1991.

[3] P. J. Ashenden. The Designer’s Guide to VHDL, Volume 3, Third Edition (Systems on Silicon) (Systems on Silicon). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2008.

[4] R. Azuma. A survey of augmented reality. Presence, 6:355– 385, 1995.

[5] R. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Julier, and B. MacIntyre. Recent advances in augmented reality. IEEE Comput. Graph. Appl., 21(6):34–47, 2001.

[6] E. R. Babbie. The Practice of Social Research. Wadsworth Publishing, Feb. 2009.

[7] Cellagames. Augmented reality tower defense, Jan. 2010.

[8] P. M. Chisnall. Marketing research; analysis and measurement [by] Peter M. Chisnall. McGraw-Hill London, New York,, 1973.

[9] C. Cooke. Charades: A Classic Party Game. Meadowbrook, Mar. 2003.

[10] Google. Android open source project, jan 2010.

[11] G. F. Guimaraes, J. P. S. M. Lima, J. M. X. N. Teixeira, G. D. Silva, V. Teichrieb, and J. Kelner. Fpga infrastructure for the development of augmented reality applications. In SBCCI ’07: Proceedings of the 20th annual conference on Integrated circuits and systems design, pages 336–341, New York, NY, USA, 2007. ACM.

[12] Hasbro. Guess who? instructions, Jan. 2003.

[13] B. Kisacanin, S. S. Bhattacharyya, and S. Chai. Embedded Computer Vision. Springer Publishing Company, Incorporated, 2008.

[14] A. E. Lab. Arhrrrr!, Jan. 2010.

[15] T. Lorenzen and H.-L. Chang. Mastermind c : a predictor of computer programming aptitude. SIGCSE Bull., 38(2):69– 71, 2006.

[16] Parrot. Parrot ar.drone, Jan. 2010.

[17] PrintActivities. Rules for hangman word game, Jan. 2008.

[18] C. B. Raimo van der Klein and M. Lens-FitzGerald. Layar mobile augmented reality browser, Jan. 2010.

[19] B. Schwald, B. D. Laval, T. O. Sa, and R. Guynemer. An augmented reality system for training and assistance to maintenance in the industrial context. In In The 11th International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision’2003, Plzen, Czech Republic, pages 425–432, 2003.

[20] M. Sevenster. Battleships as a decision problem. ICGA Journal, 27(3):142–149, 2004.

[21] M. M. Software. Wikitude world browser, Jan. 2010.

[22] J. M. Teixeira, V. Teichrieb, and J. Kelner. Embedded Augmented Reality: Finding an adequate escape pod to real time augmented reality applications. LAP Lambert Academic Publishing, Dec. 2009.

[23] D. Wagner, T. Pintaric, F. Ledermann, and D. Schmalstieg. Towards massively multi-user augmented reality on handheld devices. In In Third International Conference on Pervasive Computing, 2005.

[24] wHow.com. How to play pictionary, Jan. 2010. [25] Wikipedia. Guessing game, Jan. 2010.