c175. ICG2000 Muscat, Oman 99.12.16

 

 

ASYNCHRONOUS LEARNING/DESIGN AND USER PERFORMANCE USING A WEB GROUP DECISION SUPPORT SHELL WITH AN OPTIMAL COMPLEXITY FRAMEWORK - MODELS FOR DESIGN OF BENIGN URBAN WATER SYSTEMS

by

William James, D.Sc, P.Eng

Professor of Water Resources Engineering

University of Guelph, Canada N1G2W1

E: wjames@uoguelph.ca; W: www.eos.uoguelph.ca/webfiles/james; F: +5197672770

 

ABSTRACT

Web use of a group decision support system (GDSS) for design of benign urban water systems is described. Using distributed, shared information systems on the web, the PCSWMMGIS shell integrates the three design packages SWMM, EPANET and WASP5 in an Internet learning/design environment, and is expressly developed for improving user performance. An aggregation wizard for optimizing model complexity is described. Complexity may vary from coarse to fine. Among other GDSS tools, sensitivity and error analysis, and on-line animated tutorials tools are included. Using this framework, graduate students from three countries participated in a web graduate course on the web. Experience shows that achievable modelling accuracy depends more on user quality than it does on model structure. As distinctions between instruction, learning and design (work) disappear, a GDSS that facilitates demonstrably improved user performance is desirable. Asynchronous Internet learning environments are now ubiquitous, and an important test bed for evolving design technologies.

INTRODUCTION

SWMM is used as the glue for the development of several new ideas presented in this paper with an complex title. For the benefit of readers unfamiliar with the subject, and by way of introduction, some background information is first given on the EPA Stormwater Management

Model SWMM (herein, reference to SWMM relates to version 4.4GU of December, 1999). Development of SWMM from 1969 to date is described in Introduction to the SWMM Environment, Chap. 1 in New Techniques for Modeling the Management of Stormwater Quality Impacts (James, 1993). Development of SWMM, its ancestry and its continuing support is probably unique. Intermittent support by the USEPA has been augmented by research groups at several different universities, including those of the author, the University of Florida, and Oregon State University, and by engineers at agencies and in consulting offices, who have spasmodically contributed ideas or more materially to the evolution of SWMM. In many ways, defining SWMM has thus become like nailing the proverbial jelly to a tree. What was unusual, even at the time, was that SWMM was a major EPA funding effort devoted solely to hydrologic software development (another example was the conversion of HSP to HSPF, funded by EPA CEAM). From 1974 there was close collaboration with Canadians - notably the Ontario Ministry of the Environment, Environment Canada and several consultants and academics, also an unusual if not unique effort (James, 1978a,b). However, SWMM itself remains a cumbersome accumulation of batch-oriented FORTRAN code. PCSWMM was the first user-friendly shell and personal computer version. It has been distributed with improved documentation by the writer (URL: www.CHI.on .ca) since 1984, and for the Windows environment since 1993. As of Nov. 1999, it was used in about 800 design offices and 70 Universities, in about 30 countries, predominantly North America.

Probably the SWMM environment is a natural consequence of active participation in scientific, technical and engineering conferences, symposia, seminars, workshops and other meetings. Besides workshops and short courses given by (i) the University of Florida, (ii) the USEPA, and (iii) the writer, the more common meetings include irregular six-monthly user group conferences in the US and Canada. Recently, in the 1990s, meetings have been held annually in Toronto. Proceedings of these conferences are generally difficult to locate in libraries or elsewhere, so the author has made available (Biblio’98, also as part of PCSWMM) abstracts of the papers. Papers range over many topics of interest to users of SWMM, and nicely cover the changing emphasis over the past two-and-a-half decades. Topics change from concerns with water flow quantities in a remote-batch-mainframe environment, to interdisciplinary ecosystems concerns in the still-evolving networked-workstation design environment described in this paper. One of these topics is the development of shells like PCSWMMGIS.


PCSWMMGIS

PCSWMMGIS was written for the web, and is expressly designed to improve user comprehension of the underlying physical processes. It has depended on the work of many of the 55 graduate students listed on the writer's home page. Of these, the work of Dunn (1986) and Robinson et al. (1984) are typical. Being a decision support system for SWMM, PCSWMMGIS adds GIS graphical utilities to PCSWMM, thus providing a large array of file management, data file creation, output interpretation, model calibration, and reference tools. Although focussed on the needs of the urban water systems modeler, an open approach has been taken to provide a wider level of flexibility and power. Users can develop their own in-house modeling environments using the extensive selection of plug-in tools and seamless integration of any external processes (programs, batchfiles, macros, etc.). It is distributed with the latest BETA version of SWMM from Oregon State University. Advanced file management methodology has been enhanced for use with data distributed over local area networks (LAN's), Intranets and the Internet. The shell simplifies the organization, connectivity and manipulation of the many files generated and used by SWMM. Data-aware objects represent groups of related database-oriented files. Object images depict the type of files represented. These objects can be arranged logically in folders, connected to other objects, and cut/copy and pasted between folders. Project information management is provided by means of note objects, which can store text in ASCII, HTML or any document format. File extensions are handled automatically by PCSWMM, and can be specified by the user. Existing SWMM files can be easily imported into PCSWMM, and/or new data files can be created from user-specified template files. Some attributes relevant to user-performance improvement are:

  1. An extensive, online, indexed help file which provides a variety of information. Step-by-step instructions are available for all PCSWMM procedures. Known trouble spots are highlighted for quick explanations. PCSWMM also provides complete online help for all aspects of SWMM data file development for any module of SWMM. Up-to-the-minute help is available on-line from the active desktop through a WWW site and email services.
  2. Tools take the form of external processes that can be plugged in to PCSWMM with the Tools Registry. This enables a completely flexible modeling environment created from third party and in-house tools, as well as the powerful tools bundled with PCSWMM.
  3. OpenURL enables access to email, web pages, ftp, and telnet from Internet Shortcut objects, the Tools menu, or any URL address found in any object file. It also downloads the most recent version of SWMM with one mouse click, includes links to related information or data sources on your intranet or the internet in your input data files or note files, and/or provides easy modeling coordination with remote teams. Of course this requires a WWW browser and Internet access (can be dial-up). OpenURL comes with a help file for instruction on usage.
  4. Biblio'98 provides instant access to reference information on 4500 papers given at SWMM users meetings and conferences. In 1998, the database included over 3500 searchable abstracts with two powerful search engines, and an updated user interface.
  5. Sensitivity Wizard is a powerful tool for gaining insight to the dominant processes of a particular model, and how they can change through an array of input functions (i.e. different rainfall intensities and durations). Virtually any number of RUNOFF parameters can be tested automatically, reducing a virtually unmanageable task to a few mouse clicks. Each chosen parameter is tested at five positions over user-specified ranges of uncertainty, the SWMM runs are automatically batched and results compiled from the output files. Sensitivity results are presented in a variety of ways, including both non-linear sensitivity gradients, and ranked mean sensitivity gradient plots, and can be easily customized and printed or copied into reports at high resolution.
  6. Calibration Wizard generates observed vs. computed plots, complete with an array of evaluation functions, for any objective function. The program reads in a user-generated list of observed/computed data points, and the resulting plots can be customized and printed/copied into reports.
  7. Tutorials with extensive online help, Getting Started video tutorials and web support ease the learning curve.
  8. The GIS facility is designed for creating and/or editing node/conduit networks for the SWMM model. Its primary application is to link a GIS/IMS with SWMM (downwardly compatible), accomplished through a three-step process:

GIS features include:

OPTIMAL MODEL COMPLEXITY

Model complexity is a measure of both the number of processes activated and the number of sub-spaces (pipes and facilities) coded in the datafile. It is thus related to the size of the datafile, excluding the driving time series. Disaggregation is the intellectual procedure of dividing a real system into its theoretical component processes and sub-spaces. Aggregation is the reverse, the procedure of averaging a larger number of processes and sub-spaces into a smaller number of lumped sub-spaces and processes. The model is said to be less complex.

Once the model data is in PCSWMMGIS, it can be edited graphically, through the local tables, and/or with help from the Aggregation Wizard (which intelligently reduces model complexity). For reducing model complexity PCSWMMGIS can be used on its own, without an existing GIS/IMS source: new data can be entered graphically or through the local database tables, and/or imported from existing SWMM input data files and other sources. Other features of PCSWMM GIS include:

  1. Model output visualization and analysis directly within PCSWMMGIS (i.e. thermometer style dynamic plots etc.);
  2. Written as a standalone tool, the software interfaces directly with the underlying database(s). ESRI, MapInfo and AutoCAD layers/themes can all be displayed. Look and feel is most similar to ArcView (although MapObjects were not employed). Data is extracted directly from the GIS's underlying database (MS Access or ODBC compliant) using SQL queries and setup in PCSWMM GIS's internal database (MS Access) for 'data tweaking' into a suitable model (i.e. element aggregation, etc.). External access to the PCSWMM GIS internal database is also possible. A *.bmp or meta-file can be used as a background with completely accurate scaling. In fact as many tiled or layered images as desired may be loaded. Supported raster file formats include: .TIF, .BMP, .JPG, .PCX, .DIB, and .TGA. Other (vector) layer formats supported are: MapInfo MIF, MicroStation DGN, AutoCAD DXF, AutoCAD DWG, and ESRI Shape File. Vertical detail is stored in the Extran layer of course, and PCSWMMGIS supports real world coordinates (and any other coordinate system). Its primarily focus is as an interface between a GIS/IMS and SWMM. It as assumed (though not necessary) that most model development will take place within the PCSWMMGIS and PCSWMM environment. However, PCSWMMGIS can be used without a GIS/IMS as a graphical plan view editor (with or without backgrounds) for SWMM or PCSWMM. Its interface can be used to automate data input such as watershed areas, land use, and soils related parameters such as infiltration rates (if you have GIS data with these attributes). Node, conduit, and subcatchment data is pulled from the underlying GIS database (ODBC) into an intermediate database for processing into a useful model. This data is then exported to a SWMM input file (Runoff, Transport or Extran).
  3. Apart from the Aggregation wizard, which calculates equivalent parameters when aggregating two or more conduits to a single conduit, PCSWMMGIS does not calculate/estimate any parameters. Also PCSWMMGIS cannot determine input parameters directly from shape files - just from the underlying attribute database. True GIS (ArcView and/or ArcInfo for example) products are best suited to this kind of analysis. Once these attributes have been determined and exist in the GIS, PCSWMM GIS facilitates the data extraction and model development.

In the next section we consider the wider context of design and user-performance improvement in the evolving Internet environment.

WIDER DESIRABLE WEB ATTRIBUTES OF A DESIGN/LEARNING GDSS

Ostrowski and James (1997) make a case for web-based GDSS in environmental design. From the technological point of view a suitable general definition of DSS is given by Osmond (1997): DSS are computer based information systems that are designed with the expressed purpose of improving the processes and outcomes of human decision-making. Ostrowski and James see a distinct trend in urban drainage away from standard techniques like central, separate or combined sewer systems, towards future strategies and technologies that seek integrated optimum individual solutions on the house or neighbourhood scale. However a more complicated planning process is a major obstacle to the introduction of decentralised solutions. It is far more difficult to convince a group of individual citizens of the ecological and socio-economic advantages of a particular design than it is to create the design in the first place. Transparency and continuous information are required for its acceptance. Computer network technology like the web is far ahead of decision making techniques for group decision-making. Despite a high technological standard for water supply, drainage comfort and sewage treatment of traditional urban water systems in western countries, it has become evident during the last decade that traditional technology approaches also cause manifold ecological and socio-economic problems. Even in these continuously ageing water systems, the design problem is to modify or replace infrastructure after it has reached its useful life span. Recent proposals for urban water infrastructure replacement mainly aim at decentralised, individually tailored solutions, which in turn require efficient tools for decision-making among small groups, neighbourhoods and communities.

Watkins and Mc Kinney (1995) summarise more than 20 years of decision-support-systems (DSS) in water resources planning and management. The introduction of locally and regionally optimum solutions requires the participation of citizens and citizen groups concerned, who must be informed about impacts on their property, about their wider neighbourhood environment and about the costs of the proposed designs. A high degree of transparency is compulsory for the general acceptance of severe changes. Formation of spatially distributed parties involved in decision-making by using the web seems to be a timely solution. What are the requirements and structure of such a GDSS? Ostrowski and James (1997) researched literature and numerous websites dealing with DSS and GDSS in water resources development and management, and the results are posted on the web (see "STEELDSS"). Lemons & Brown (1997) presented a recent important contribution to the topic, analysing the information available on Sustainable Development (SD). An important objective is to enable the scientific and technological community to make a more open and effective contribution to the decision-making processes concerning environment and development. Olsen and Courtney (1992) discuss development and assembly of GDSS, and Gordon et al. (1996) presents a web-based group decision system.

For the design of such a GDSS, Ostrowski and James (1997) propose several modules:

POAM would be a list like SWMM-USERS, WASP-USERS and EPANET-USERS. Interested individuals can subscribe to the module, screen an archive of opinions and form sub-groups. Necessarily those professionally involved in the urban water system considered must subscribe to POAM to be continuously informed about the ongoing discussion. Questions of information, communication and information dissemination are crucial (Courteau, 1996). The Internet can become a strategic tool for consolidating these functions.

INSYST is the general knowledge base of the GDSS. It could contain:

EDMOS must contain modules for all stages within the pre-school, school and university systems as well as adult education. Courteau (1996) stresses the importance of public education - environmental education means life-long learning. Curricula for elementary and secondary schools are necessary. Environmental issues must be included to a higher degree in undergraduate and graduate college and university courses, not only in environmentally oriented subjects. The Web can become the medium for teaching courses to everybody, even degree courses. First experiences with web-based courses on water systems as described by James and Ostrowski (1997) are given below.

BMPM allows local optimum urban water system planning to choose from a variety of options and alternatives. Costly technical standard solutions might still be feasible, but should not be generally accepted and applied. A full economic, socio-economic and environmental assessment must be the basis of decision-making. The availability of such a survey of available approaches is most important, because limited textbooks and other printed material are available.

MEXSY relies on basic data, which might partially be stored in the information module INSYST. The importance of modelling expert systems in environmental planning is unquestionable. Yet, their stage of development and application is mostly limited to planners. Due to the complexity of such models the development of generally accessible and understandable model systems takes time and requires basic knowledge, which is again linked to the educational modules. Models also play a big role in the educational process. The complexity (degree of abstraction) must be linked to the level of education.

The above five ‘service modules’ would be grouped around EVADEM, the kernel module for decision-making. Information technology is not the bottleneck, but other important factors that influence decision-making. Some are listed in Table 1, which shows that anthropogenic impacts on natural resources can be considered at different temporal and spatial scales. In the context of SD it is important to differentiate between local short-term reversible and non-consumptive impacts and long-term global irreversible consumptive impacts. Tomalty and Pell (1994) analysed additional requirements to allow public participation in assessment and decision-making for Canadian cities. Their results are summarised in Table 2. Tables 1 and 2 contain mainly qualitative aspects and evaluation criteria, which are difficult to evaluate and compare. For this reason it is essential to quantify criteria which can be expressed in numbers. Examples of such criteria are given in Table 3.

Table 1: Important factors influencing decision-making

Influencing factor

Type or category

Resources

Human (number, age, education, health)

Financial (gross national product)

Natural (air, soil water, energy)

Temporal scale

Eternity>long-term > 1 generation

1 generation> mid-term > several months

several months > short-term > several days

several days > acute

Spatial scale

Global (the planet including its atmosphere)

Continental (subcontinents to continents)

Regional (stream catchment, large cities)

Local (small catchment, small communities)

Severity of impact

Reversible (within mid-term range)

Irreversible (within long-term range)

Consequence of use

Consumptive (distinct change of quality and quantity)

Non consumptive (no change of characteristics)

Objectives

Economic (stability, growth)

Ecological (deterioration , no further degradation, improvement)

Socio-economic (equity, polluters pay, democratic)

 

Table 2: Community based SD planning (Tomalty and Pell, 1994)

Principles

Requirements

Community

Is the decision process explicitly community- based?

What ongoing consultation with community residents have been included?

Does the planning process address community concerns?

What rationale is there for the process in the stated concern of community residents?

How does the process include consideration of identified social issues?

Equity

Is the process democratic?

Does the process promote equality of opportunity

Does it clarify or make explicit its basic values and beliefs?

Are those values and beliefs consistent with a sustainability perspective?

Self-

Determination

Is the process open and accessible to those who will be affected by it?

Does it promote self-management at a local level?

Integration

Is the process systematic in orientation?

Is the process synergistic, taking into account a variety of perspectives?

Does the process acknowledge and account for inter-dependencies among social, environmental and economic well being?

Balance

Is the process cyclical and reflective?

Does it incorporate ecological principles, such as geographical

Boundaries, cross-generational timeliness, and social and physical carrying capacity?

Diversity

Is the process inclusive?

Can it embrace apparently opposing views, such as persistence and change?

Does it anticipate and expect divergent points of view?

Does it provide mechanisms for achieving consensus or otherwise

resolving disagreement in situations where differences of opinion occur?

Table 3: Quantifiable evaluation criteria

Economic

Environmental

Social

Leakage/km pipe

Water quality with emphasis on nitrate, phosphate & pesticides

Number of disconnections for non-payment

Unit cost of supply and disposal

Number of reported pollution incidents

Interruptions of supply

Number of water meters installed

Total sediment and pollution load delivered to oceans

Drinking water quality

Energy use / unit

Changes in the extent of wetland habitats

use of water per capita

Interest rates

Groundwater fluctuations, reduced baseflow

Recreational usage

 

THE FIRST INTERNATIONAL WEB-BASED GRADUATE COURSE ON BENIGN, INTEGRATED URBAN WATER SYSTEMS

It seems to be mandatory that watershed management plans be integrated. Local micro-scale impacts can be integrated and averaged and be input to meso-scale GDSS. Although some approaches have been developed much work needs to be done to build and test a meso-scale GDSS. Most of the concepts discussed above in this paper were put to the test in summer of 1997, when James and Ostrowski (1997) jointly offered a graduate course. Of the eight graduate students in Germany, Canada and Brazil who started, six completed the course.

General objectives were to become adept at models that analyse urban water systems, the model applications being constrained by eco-sensitive philosophies. Hardin’s Tragedy of the commons, in which community assets are inevitably depleted to the advantage of individuals, was a starting point. Questions addressed included: how can water systems be developed such that there is no net increase in human population? Models studied could be used for design of less unsustainability. The models estimate flow and pollutant quantities and optimise conveyance, storage and pollutant removal structures required for storm, sanitary and combined sewer collection systems, potable water distribution systems, and the management of the impacts of such systems. Models covered included WASP, EPANET, PCSWMM, and SMUSI. Twelve modules were designed such that students could elect to cover all four component models (generalist track) or to specialise in either of the two urban drainage models (PCSWMM or SMUSI) using all the time available (specialist track). Students were expected to become independently competent on the web, Internet, HTML and perhaps Java scripts. Using Internet utilities like email and the course list server, they developed their own websites, and themselves evaluated colleagues’ submissions. At the end of the course the German students visited Canada.

In essence, the course, all of which was run on the web, was very practical, and fully exploited the entire global water environmental resource. Time of teaching, learning, and evaluation and location of teachers and learners was utterly asynchronous: students fully controlled the time and location of their effort. Instructors developed material and the website in June, students learning participation took place in July and August, and both groups conducted the evaluations in August and September. Not foreseen was the effect of the rapid expansion of the web during this interval. Evaluations were based on web pages developed and were conducted by the participants themselves. All pages, viz. the instructional materials and the learning products as well as all evaluations, were, and remain, publicly posted. Students posted their summaries of their colleagues' evaluations together with their original assignments on their own home pages. In the end, because of the unexpected high workload, caused by the web’s continuing expansion, only eight modules were provided, and in most cases about six were completed. The website for the course is at: www.eos.uoguelph.ca/webfiles/james/wj661HomePage.html

Student webpages can be accessed at that site by scrolling down the left-hand toolbar to Class list and clicking on a web button for one of the students. For Andrew Chan, visit: http://home.istar.ca/~andychan/swmindex.htm

As delivered, the instructors developed the web pages some two months ahead of students. Gradually it became clear that web-based learning may be rather slow, due to language differences, communications band-width limitations, the rapid expansion of the web itself, and the difficulty of evaluating the worth of the wealth of web-based information. Full course content is available on the web, but below is a list of the modules and their content as finally developed:

0: PRELIMINARY ORIENTATION AND ORGANIZATION
1: INTRODUCTION
2: IMPACTS OF URBANIZATION ON AQUATIC ECOSYSTEMS
3: POLLUTION CONTROL OPTIONS
4: URBAN INFRASTRUCTURE
5: STRUCTURE OF SELECTED MODELS
6: MODEL DECISION SUPPORT
7: APPLICATION AND SENSITIVITY ANALYSIS OF THE WASP EUTRO5-SUBMODEL
8: APPLICATION AND SENSITIVITY ANALYSIS OF SWMM AND SMUSI MODELS
9: SANITARY SEWER COLLECTION SYSTEM MODELS
10: WATER DISTRIBUTION MODELLING

Student feedback uncovered concerns regarding

    1. the sheer quantity of work,
    2. differences between workload expected in the different academic environments (European students expected workloads of ca 60 hours whereas Canadian expectations were 120 hr),
    3. difficulty of working across three languages,
    4. inexperience in evaluating the value or worth of information on the web,
    5. apprehension about creating and publishing new information on the web, and
    6. the lack of human contact (especially instructor-student).

Perceived benefits include the establishment of international collaboration on graduate assignments, other further international contacts from their required Internet activities, web-literacy, and the revelation of philosophical arguments involved in the development of urban water systems. Extraordinary was the realisation that these were the self-same concerns and gains enjoyed by the two instructors. Thus students and instructors alike found the course to be a large amount of hard work, and that we actively learned through the authorship of instructional instruments. Truly, it can be said of this experience, more than in our many years of traditional teaching, that the distinctions between learning, teaching and work had vaporised.

CONCLUSIONS

Based on the experience gained in a generation of development and use of earlier shells, and their application at over 50 professional design workshops, as well as in graduate and undergraduate credit courses over that time, a wholly new design shell for the U.S. EPA Stormwater Management Model SWMM was developed. The new shell was designed as a web-oriented GDSS to facilitate the teaching and learning of SWMM in design offices, in international SWMM seminars and workshops, as well as in graduate and undergraduate courses at the University of Guelph. Subsequent experience helped design a system that speeds classroom learning times, and maximizes use of available resources, by eliminating the most-expected errors. Classroom use of PCSWMM in settings of 100 or more students, has demonstrated significant improvement in learning times, and reductions in the most prevalent run-time errors.

The web, as opposed to other attributes of the Internet, can offer a very wide diversity of design aids and instructional media and opportunities. Because of its continuing exponential growth, the web always provides a much larger and richer information resource than anyone expects. Students need to be taught to be discriminating in evaluating good web information. Asynchronous delivery of the diversity of design and instructional materials and products in this Internet information environment is unequalled by any other technology or media known to the author.

Philosophical issues such as those raised by questions of the tragedy of the commons, human population control, ecosystems destruction, and the inherent unsustainability of the western value system, are readily dealt with in a web environment. Overall, the experience showed that both the teaching and learning effort in such a course could be formidable. Asynchronous learning environments, perhaps in common with others, can remove the distinctions between learning, teaching, and design (work). The question of optimal complexity for the evolving design detail, and gradual user sophistication is only now being addressed. The web instructional setting provides an important test-bed for development of group design decision support systems, especially where the desiderata include user performance.

REFERENCES

Courteau, I. (1996). Environmental education for sustainable development, ECODECISION, Autumn edition, p. 5
Dunn, A., 1986. "Feasibility of a Real-Time Auto Sensitivity and Error Analysis Framework for Fuzzy Stormwater Modelling on Personal Computers", M.Eng. Thesis, McMaster U., May 1986, 130pp.
Gordon, Th. (1996). Zeno - A mediation system for spatial planning, Proceedings of the ERCIM-Workshop on CSCW and the Web, Sankt Augustin, Germany, February 7
James, W., 1978a. "Fast SWMM", Can. Soc. of Civil Eng'rg, Conf. on Computer Applications in Hydrotechnical and Municipal Eng'rg, Toronto, Ontario, pp. 325-341, May 1978.
James, W., 1978b. "A Pre- and Post-Processing Program Package for the Stormwater Management Model", Stormwater Management Model User's Group Meeting, Annapolis, Maryland, EPA 600/9-79-003, pp. 218-233, Nov 1978.
James, W. 1993. "Introduction to the SWMM Environment", Chap. 1 in New Techniques for Modeling the Management of Stormwater Quality Impacts (W. James, Ed.) pp 1-28.
James, W. & Ostrowski, M.W. (1998) Lessons learned during a first international graduate course on sustainable urban systems modelling using asynchronous-learning on the web. UDM ’98, London, September 1998
Lemons, J., Brown, D.A. (1997). Sustainable development: Science, Ethics, and Public Policy, Kluwer Academic Publishers, Dordrecht, The Netherlands
Olsen, D.L., Courtney Jr, J.F. (1992). Decision support systems and expert systems, Macmillan Publishing Company, New York
Osmond, D.L., Gannon, R.W., Gale, J.A., Line, D.E., Knott, C.B., Phillips, K.A., Turner, M.H., Foster, M.A., Lehning, D.E., Coffey, S.W., Spooner, J. (1997). WATERSHEDSS: A decision support system for watershed-scale nonpoint source water quality problems, J. of the Amer. Water Resources Assoc., April 1997, p. 327-341
Ostrowski, M.W. & James, W. (1997). The concept of sustainability of water resources systems - A group decision support system STEEL for urban drainage (http://www.tu-darmstadt.de/fb/bi/wb/ostrowski/steel-dss/index.html)
Robinson, M.A., James, W. and Unal, A., 1984. "Continuous Variable-Resolution Stormwater Modelling on a Microcomputer Using a Central Hydrologic Data Base Manager", Proc. of the Can. Hydrology Symp., Quebec City, Quebec, Jun 10-12, 1984, published by National Research Council of Canada, NRCC No. 24633, pp. 499-514, 1984.
Tomalty, R., Pell, D.(1994). Sustainable Development and Canadian Cities: Current Initiatives, Canadian Global Change Program Incidental Report Series No. IR94-1, Ottawa
Watkins, Jr., Wm Mc Kinney, D.C. (1995). Recent developments associated with decision support systems in water resources http://www.earth.agu.org/revgeophys/watkin00/watkin00.html)