dbnet.home Bookmark and Share

MEET FEATURE REQUIREMENTS, SCHEDULE, and BUDGET! Consulting available to teach your organization to apply this methodology and tools for powerful quantitative management of your business processes.

  • Instruction of your staff at your site, using courseware, application example, and a functional Framework.
  • Mentoring your key staff on an ongoing or periodic basis, on these tools or on Quantitative Program Management.
  • Contracting or Employment in your organization on specific topics.

Brochure  ...  Contact for Details

Check out my YouTube Channel: Power Operational Intelligence

Now Live! Overview, Structure, Task Data, Table Design, SQL Server, and Re-Linking now showing.

Video courses covering material on this website, and more, are presented in the playlists.

Code snippet links at YouTube Code Snippets. Twitter at @poweroperation1, #poweropi, #poweroperationalintelligence.

Subscribe on YouTube, and click the "Notification" Bell icon to be notified as content is published.

SQL Server Security Structure

An article on this website, organizing-program-communication, makes the case that solid communication is a foundational element of Program Management. This was probably already obvious, or hopefully the article made a convincing case! Part of that communication includes Program planning and logistics, which affect the work of most people working on the Program. Breaking-up Program data, particularly for logistics, into separable problems (e.g.: use/allocation, product structure, sequence/dependency/deliverable/schedule, material/build, acquisition/distribution, finance) leads to a better quantitative description of Program elements, lessening breadth of extraneous knowledge needed to tackle each problem. JOINing those data elements appropriately provides very strong leverage, and retains accuracy. Computers can do that work, with the resulting hierarchy of data providing quantitative information and program coordination for decision-making and direction un-imaginable compared to use of spreadsheets to manage such data, common even today.

You'll find that as Program size and complexity increase, access to PM data by more people provides needed leverage. The cornerstone of a shared data accessed by numerous remote Clients is Database Security.

Beyond collection and reporting of more data, some of the increased data involves more sensitivity; a shared system brings with it broader complexities of access and of protection of data and of system integrity. Access must be allowed from local and remote clients known to the database system only from each connection request. With access by personnel in more roles on the program, and particularly from remote clients, appropriate access validation is needed for access to data that may be sensitive: proprietary, or organizational, or personnel-related. Provision of access, and of permissions to see and update particular types of data and to perform operations, is at the heart of using a shared data system. Data security capability is provided by the Database Engine. OpISQLServerSecurity is a tool that puts those capabilities to use for PowerOpI applications.

First, for Microsoft SQL Server, a must-read article describing principles of security for that system: SQL Server Security Overview of SQL Server security structure.

That article identifies several security Principals: Server Logins, Server Roles (Fixed, and User-Defined), Database Users, Database Roles (Fixed and User-Defined), and Application Roles. These Principals can be granted Permissions of access and operation. Logins, Database Users, and Roles can be also added as members to Roles, creating a hierarchy of user capabilities. Permissions and hierarchies are defined by OpISQLServerSecurity and described below, for use with the PowerOpI tools for Shared and Remote data access.

OpISQLServerSecurity implements a set of Server Logins, Server Roles, Database Users, Database User Roles, and data-organization Schemas. These are defined in Tables in OpISQLServerSecurity\ToolSource.xlsm, and actuated by a set of provided SQL Scripts executed from SQL Server Management Studio (SSMS).

PowerOpI default User types implemented for both Server Login and Database User

  • OpISuper: Everything. Project Databases; Roles, Logins, DBUsers, Domains.
  • OpIManager: Assign Project members as DBUsers of a Project Database.
  • OpIDataOwner: Update Project Data in Project Database.
  • OpIClient: Reports from Project Data in Project Database.
  • PMLogin: Access to schema with sensitive data: fin, hr, mgr, org.

Schemas provide file groups within Project Databases, allowing access by Database Users with particular Database Roles that GRANT permissions within a Schema:

  • dbo: Default SQL Server data access.
  • fin: Project finance.
  • hr: Project personnel, accessible by HR personnel DBUsers only.
  • mgr: Management information such as project assignment etc.
  • org: Organizational information such as Org Structure etc.
  • opi: Reserved schema for embedded security tables uploaded from OpISQLServerSecurity.
  • It is intended that schemas set up by OpISQLServerSecurity for sensitive data should deal only with data related to management of the Program to be managed. This would not be a primary data source for HR or Finance or Management for the organization as a whole. Data would be limited to that which is consequential to management of a specific program. Typically organizations responsible for broader organizational data systems may provide Views or Permissions relevant to roles and positions. See also Principle of Least Privilege.

Security Role Schema shows hierarchy and intent of PowerOpI Database User Roles

  • SecuritySchema

These figures show detail of Permissions available across all of SQL Server, and subset groupings of permissions via Server Fixed Roles and Database Fixed Roles:

  • Click image for SQL Server permission descriptions and poster image download:
    images/vbaConnectionImages/database-engine-permissions.png
  • Server-Level Fixed Roles showing Permissions
    images/vbaConnectionImages/permissions-of-server-roles.jpg

SQL Server Security For Small Management Structure

PowerOpI tool instance SQLServerRemote includes Server Login and Database User configuration scripts for operation among a tight group of Program management participants. This is in contrast with Programs that may need to manage detail information collected from numerous sources and data owners, with more broadly-needed information and report dissemination among a larger group of individuals.

Security and access roles are set up for Server Logins and Database Users identified as ProgramManagement and ProgramManagement2.

An example Small-Management Program might involve Program Management responsible for the Program structure and its operation; Engineering Management responsible for program development; Product Management responsible for Business Unit business plan, product portfolio positioning, and customer interaction; Development Operations responsible for Business Unit budget and personnel; Corporate Finance management responsible for funding and accounting; Manufacturing Operations responsible for production and development Manufacturing tasks by CM and in-house Manufacturing; and Supply Chain responsible for supplier and contract management, and material acquisition.

Management of data creation, management, and distribution for such a program is characterized by grant of limited Fixed Roles at Server level and Database User level to individuals performing such roles. Such empowerment is satisfactory for highly-responsible and trusted individuals well-known among themselves and related management structure. Fixed Roles open the system fairly broadly, permitting full data structure management capabilities. Use of Fixed Roles rather than of Grant Roles is simpler, requiring less detailed expertise and management of the data system. Individuals involved are empowered to handle many situation within the data and its containing database.

Details of structure for security of tight-management programs is described on page Remote.

SQL Server Security For Shared Remote Systems

As people and management become more numerous and distributed, access to some data becomes more sensitive . People responsible for sensitive data have less visibility of how it may be used throughout the organization as management and physical structure broaden. For such broader situations, best practice is follow the Principle of Least Privilege, providing only Permissions necessary to perform assigned Program tasks, throughout the organization.

Sensitive Program information may be inadvertently used, modified, or disclosed. Imagine an article on an un-announced revolutionary program, showing up in public press and affecting earlier competitor response. Or, data could be changed incorrectly due to mis-understanding, knowledge, or even mal-intent. A database update affecting work of many people, incorrectly done could cost significant effort, time, local or field rework, or funding to correct. Personnel issues as organizations re-structure, management changes, or even work execution. Program finance could involve cost of staffing.

Net, program data may contain sensitive information that should be intentionally managed, commensurate with need for access. So for small-management Programs, Fixed Roles open system security to a small, trusted group to make capabilities available. For broader, distributed programs, more detailed security is advisable, implementing structured access and capabilities among users that support their tasks while protecting sensitive information.

Download OpISQLServer Security

The tool is functional as described on this website. It contains full versions of all Tables described here, and contains the full SQL Scripts also described. The Tables include drop-down selections for specified keywords or values. Explanation and usage notes are provided for each Table.

Default security Principals and Permissions are specified via Tables in OpISQLServerSecurity\ToolSource.xlsm, described by the following illustrations. Table content is set up by an OpISuper admin, and security Scripts provided are executed to update SQLServer Database Engine system tables to actuate the described parameters. Organization of the Scripts and their location within ToolSource.xlsm is described as follows:

Security Script Organization

  • SecurityScriptOrganization
  • Common Variables table sets up some variables used within tblOpIRolesGRANT and FIXED and should be set up first.
  • The Scripts are enumerated purposefully, 1-through-5. In Setup, Scripts should be executed in the order of enumeration. Security_1 Scripts (GRANT and FIXED) create Role predicates for Server Logins set up in Security_2 Script. Security_2 ServerLogins are predicates for Security_5 Script that creates DBUsers.
  • Security_3 Script creates SCHEMAs, and Security_4 Scripts (GRANT and FIXED) create Role predicates for DBUsers set up in Security_5 Script.
  • On tear-down of any parameter (e.g. deletion of SvrLogin or DBUser or Role, Scripts should be executed in REVERSE: 5 before 4 etc. The Scripts will delete dependent table entries, but manual tear-down of known dependent entries will help prevent table clutter of entries with unknown or obsolete purpose (which could potentially contribute to security or integrity issues in undetermined fashion at a later date).

Permission Variables Setup

Set Common Permission Variables

  • tblOpIOnObj
    tblOpIOnObj

OpISQLServerSecurity Operation

Each operation of OpISqlServerSecurity involves executing a SQL Script from SSMS, which uses data from one or more Tables within OpISQLServerSecurity\ToolSource.xlsm. Note that the Tables may be pushed up into the SQL Server database engine, covered below in section Push Tables.

Before running a Script, Tables involved by that Script, shown in descriptions below, should be updated as necessary to add intended function: Server Logins, Roles, Database Users, along with configuration information for each. Then, open SQL Server Management Studio (SSMS) and from the SSMS File menu open the Script named in the descriptions in the following paragraphs to perform the operation. For example, to CREATE/ALTER/DROP a Server role, open Script SECURITY_1-CREATE-Svr-GRANT-Roles.sql in SSMS. Select the SSMS tab holding the Script for the intended operation, and click the SSMS menu bar Execute button, or right-click the Script and select Execute.

The Script will then perform actions specified in every line of the affected Tables that is in-scope of the Script operation, which is described in sections below on this page. That action includes JOINing tables together or with needed system tables, picking up field values from those Tables and creating one or more lines of SQL responsive to the field values among JOINed tables. The resulting SQL is then executed to effect the intent of the in-scope Table lines - for example, creating a new Server Role, configuring it with permissions, and adding it as a member of other specified roles. (FYI, as Server Logins are subsequently created, they can then be added as members of the created Roles to inherit their permissions and resultant capabilities.)

Tools for .sql Scripts: We're using the term SSMS, but in practice there are several tools that can be used. First, some explanation of addressing to connect these tools to the database against which a Script will execute. Then, we'll identify several tools and outline their capabilities for use with SQL scripts.

  • Connection and Addressing Open the .sql script file to run, then right-click and choose the connection dialog. These are formats for database address:
    • ServerComputerName\SQLServerInstanceName
      Example: MyServerName\SQLEXPRESS
    • ServerIPAddress:SQLServerInstancePort
      Example: 192.168.1.87:1433
      Showing a local computer, but remote IP is valid too. Port specification is optional; if not specified, connection will choose default port, typically 1433, set up in Install Instructions.
    • Server computer Firewall must be set up as specified in Install Instructions.
  • Tools to Execute, Edit, or Debug .sql scripts.
    • Microsoft SSMS (SQL Server Management Studio)
      Edit or Execute .sql script.
    • Microsoft VSCode - Visual Studio Code
      Edit or Execute .sql script.
    • Microsoft Azure Data Studio
      Edit or Execute .sql script.
    • Microsoft Visual Studio
      Edit or Execute .sql script.
      Debug .sql script.
      See Debugging SQL for installation, configuration, and operation of debug.

Each Script will only act on Table lines within its scope. The scope is typically defined by the leftmost fields of each Table, such as rolScope (Svr or DB), rolFixedGrant (Grant or Fixed), and rolActiveNow (Yes, No, DryRun). For ActiveNow fields, a "Yes" value indicates intent to execute that line; "No" indicates that line should be skipped. "Dry Run" indicates that SQL should be fully constructed but should NOT be Executed.

For all Tables, the term "ActiveNow" for each line indicates whether that line will be executed when SSMS Scripts are invoked. The term is NOT an indication of existence of the entity described on the line, rather it is an indication that action will be taken by the Script for that line when it is in scope for the Script. Scripts generally look at Scope and Verb and will use other fields as well as ActiveNow to determine action.

Each line of created SQL is echoed to the SSMS console as it is executed, and includes the Table line number involved for reference back to the source Table. Upon execution, if an error is reported by the Database Engine, the error information is echoed to the SSMS console. If the ActiveNow field indicates "Dry Run", the constructed SQL line is echoed to the SSMS console, but the line is not executed so no error will be detected by the Database Engine or echoed to the SSMS console. Nevertheless, the SQL can be inspected in the SSMS console for correct form and configuration.

Each tab in OpISQLServerSecurity\ToolSource.xlsm that contains a Table associated with a Script, notes the Script(s) that use the Table, and adds operational notes for the Table and Scripts. Some operational notes and references are also noted at the top of each SQL Script source, visible when open in SSMS.

You may find the SQL in the SCRIPTS to be interesting. It includes numerous constructs:

  • Use of local @variables, including setting value from QUERY SELECT items.
  • Use of SQL Cursor to scan through source Tables, including layering of cursors within cursors.
  • Use of Temporary Tables for intermediate data extraction.
  • QUERYs with JOINs into SQL Server system Tables.
  • COUNTing records returned from a QUERY, and determination that the count is more than zero.
  • Construction of dynamic SQL QUERYs. CONCAT function is used rather than + operator, in order to correctly handle values with NULL value.
  • Executing dynamic SQL statements using sp_executesql system stored procedure, including operations on System Tables, and reporting via SSMS Console.
  • TRY-CATCH structure to catch and handle errors as SQL statements are executed, including error reporting.

CREATE - ALTER - GRANT - DROP Server-Role

CREATE Server Roles

  • tblOpIRolesCreate
    tblOpIRolesCreate

Server GRANT-Role Permissions

  • tblOpIRolesGRANT
    tblOpIRolesGRANT
  • Using Script: SECURITY_1-CREATE-Svr-GRANT-Roles.sql
  • This Script will act on the role named in each line by field rolRoleName where rolScope = Svr, rolFixedGrant = Grant, rolActiveNow = Yes or DryRun, rolOpIVerb is not NULL.
  • This Script will act on Permissions for each such rolRoleName where grScope = Svr, grVerb is not NULL and rolRoleName = grRole.
  • This Script will act on Memberships by such rolRoleName into Role rolMemberOfRole where rolMemberOfVerb is not NULL.
  • Descriptions of Script function include syntax definitions showing how Table columns are used in the operation. The prefix lower-case characters in each name indicate the Table from which a column is sourced: rol for role definition table tblOpIRolesCreate, gr for Grant Permission definition table tblOpIRolesGRANT, fix for Fixed-Role definition table tblOpIRolesFIXED, lin for Login definition tblOpILogins, sch for tblOpISchemaCreate; usr for tblOpIUsers; and on for variable-definition table tblOpIOnObj.
  • rolOpIVerb = DROP

    For each role rolRoleName described by a line in tblOpIRolesCreate, OpISQLServerSecurity, Script first DROPs all memberships by that role, in other roles.

    Next, Script DROPs all memberships within the specified role.

    Finally, Script DROPs the specified role itself.

    -- DROP rolRolename Membership in otherRoles
    ALTER roleType -- from system lookup otherRole = SERVER
         ROLE otherRoleName -- from system lookup, thisRole is member of otherRole
         DROP MEMBER rolRoleName; -- drops membership of thisRole in otherRole
        -- Loops through all memberships in other Roles.
    
    -- DROP members of rolRolName
    ALTER roleType -- from system lookup thisRole = SERVER
        ROLE rolRoleName
        DROP MEMBER memberName; -- drops membership of memberName in thisRole
        -- Loops through all memberships within this Role.
    
    -- DROP Server Role
        DROP roleDesc ROLE rolRoleName; -- roleDesc from system lookup of thisRole
    
  • rolOpIVerb = ADD/UPDATE

    Script First CREATEs a Server Role, if it does not already exist. If a role by that name already exists, the existing role will be updated.

    CREATE SERVER ROLE rolRoleName AUTHORIZATION sysadmin;
    

    For each role named by field rolRoleName described by a line in OpISQLServerSecurity tblOpIRolesCreate, Script loops through permissions in tblOpIRolesGRANT, and executes the grVerb in each of those SERVER ROLE permissions to GRANT, DENY, or REVOKE each permission for that Role (where grRole = rolRoleName).

    Then Script executes rolMemberOfVerb to ADD or DROP membership of that role rolRoleName in the role specified by rolMemberOfRole.

    Membership of rolRoleName in additional roles may be specified by adding a line for each ADD or DROP needed.

    Note use of Table column-name variables using lower-case prefixes that indicate the source table for the variable. Prefixes are enumerated above.

    -- Syntax definition
    GRANT <permission> [ ,...n ]
         [ ON [ <class_type> :: ] securable ]
         TO principal [ ,...n ]
              [ WITH GRANT OPTION ]
         [ AS <database_principal>
        ]
        [;]
    
    -- Implemented by Script using field specifications in tblOpIRolesCreate and tblOpIRolesGRANT
    grVerb grPermission
         [ grOnObject onObj [ onObjectName | rolDatabase /* if onObjectName=onDatabase */; ] ]
         TO grRole
              [ WITH GRANT OPTION -- if grVerb = GRANT and grGrantOpt = Yes
                   | CASCADE -- if grVerb IN(REVOKE | DENY) and grGrantOpt = Yes
              ] -- if grGrantOpt=Yes
         [ AS onObjectName | sysadmin /* if onObjectName=onDatabase */
         ] -- if grAS not NULL
    [ ; ]
    

    Finally, Script executes rolMemberOfVerb to add or drop membership of role rolRoleName in upstream role rolMemberOfRole.

    ALTER SERVER ROLE rolMemberOfRole rolMemberOfVerb MEMBER rolRoleName;
    

    Membership by a created role in additional upstream roles can be added or dropped by specifying rolMemberOfVerb = ADD or DROP in additional records/lines in tblOpIRolesCreate.

  • rolOpIVerb = ALTER

    No operation is defined for rolOpIVerb = ALTER. rolOpIVerb ADD/UPDATE uses CREATE and ALTER Role management keywords internally within its functionality, but much other functionality is embedded with this opcode so these SQL keywords are not offered directly by the OpISQLServerSecurity tool.

  • References

    CREATE Server Role

    ALTER Server Role

Server FIXED-Role Memberships

  • tblOpIRolesFIXED
    tblOpIRolesFIXED
  • Using Script: SECURITY_1-CREATE-Svr-FIXED-Roles.sql
  • This Script will act on the role named by field rolRoleName in each line where rolScope = Svr, rolFixedGrant = Fixed, rolActiveNow = Yes or DryRun, rolOpIVerb is not NULL.
  • This Script will act on Fixed Role memberships for rolRoleName where fixScope = Svr (adding or dropping rolRoleName as a member of Server-level Fixed Role fixFixedRole to inherit or remove capabilities of fixFixedRole), and where fixVerb is not NULL and rolRoleName = fixRole.
  • This Script will act on Memberships in rolRoleName into Role rolMemberOfRole to inherit or remove rolMemberOfRole capabilities.
  • Descriptions of Script function include syntax definitions showing how Table columns are used in the operation. The prefix lower-case characters in each name indicate the Table from which a column is sourced: rol for role definition table tblOpIRolesCreate, gr for Grant Permission definition table tblOpIRolesGRANT, fix for Fixed-Role definition table tblOpIRolesFIXED, lin for Login definition tblOpILogins, sch for tblOpISchemaCreate; usr for tblOpIUsers; and on for variable-definition table tblOpIOnObj.
  • rolOpIVerb = DROP

    For each role rolRoleName described by a line in tblOpIRolesCreate, OpISQLServerSecurity Script first DROPs all memberships by that role, in other roles.

    Next, Script DROPs all memberships within the rolRoleName.

    Finally, Script DROPs role rolRoleName itself.

    -- DROP rolRoleName Membership in otherRoles
    ALTER roleType -- from system lookup otherRole = SERVER
         ROLE otherRoleName -- from system lookup, thisRole is member of otherRole
         DROP MEMBER rolRoleName; -- drops membership of thisRole in otherRole
        -- Loops through all memberships in other Roles.
    
    -- DROP members of rolRoleName
    ALTER roleType -- from system lookup thisRole = SERVER
        ROLE rolRoleName
        DROP MEMBER memberName; -- drops membership of memberName in thisRole
        -- Loops through all memberships within this Role.
    
    -- DROP Server Role
        DROP roleDesc ROLE rolRoleName; -- roleDesc from system lookup of thisRole = SERVER
    
  • rolOpIVerb = ADD/UPDATE

    Script First CREATEs a Server Role by name rolRoleName, if it does not already exist. If a role by that name already exists, the existing role will be updated.

    CREATE SERVER ROLE rolRoleName AUTHORIZATION sysadmin;
    

    For each role named by field rolRoleName by a line in tblOpIRolesCreate, OpISQLServerSecurity, Script loops through Server-level Fixed Roles in tblOpIRolesFIXED, and executes the fixVerb (ADD or DROP) in each of those SERVER ROLE permissions to ADD or DROP membership of fixRole (where fixRole = rolRoleName) in role fixFixedRole to inherit capabilities of fixFixedRole into fixRole (=rolRoleName), or drop those capabilities from fixRole (=roleRoleName).

    Note use of Table column-name variables using lower-case prefixes that indicate the source table for the variable. Prefixes are enumerated above.

    -- Implemented by Script using field specifications in tblOpIRolesCreate and tblOpIRolesFIXED
    ALTER SERVER ROLE fixFixedRole fixVerb MEMBER rolRoleName;
    

    Finally, Script executes rolMemberOfVerb opcode to add or drop membership of role rolRoleName in upstream role rolMemberOfRole.

    ALTER SERVER ROLE rolMemberOfRole rolMemberOfVerb MEMBER rolRoleName;
    

    Membership by a created role in additional upstream roles can be added or dropped by specifying rolMemberOfVerb = ADD or DROP in additional records/lines in tblOpIRolesCreate.

  • rolOpIVerb = ALTER

    No operation is defined for rolOpIVerb = ALTER. rolOpIVerb ADD/UPDATE uses CREATE and ALTER Role management keywords internally within its functionality, but much other functionality is embedded with this opcode so these SQL keywords are not offered directly by the OpISQLServerSecurity tool.

  • References

    Database object identifiers

    Rules for T-SQL Identifiers

    Alter Server Role

CREATE - ALTER - DROP Server Login

The default table entries will set up default Logins, with default Passwords. Use them for setup and test.

Logins, and consequently Database Users described farther below, in the following descriptions and in the downloadable OpISQLServerTool, are named in linLoginName and usrUserName for the Role that they play: OpISuper, OpIManager, OpIDataOwner, OpIClient. This is to show clearly the relationships among Logins, Database Users, Server Roles and Database User Roles. In production implementation, those Principals (Logins and Database Users) would be named for individuals, organizational positions, or the like; not by these default names.

Before you Go-Live, change these Principal names and passwords, or set up different Server Logins for production use, and DISABLE these default logins. Every user of OpISQLServerSecurity will gain the same default user names and passwords on install or over time, and could possibly use them to access your database if you have left them intact - so disable the defaults before that happens.

CREATE Server Logins

  • tblOpILogins
    tblOpILogins
  • Using Script: SECURITY_2-CREATE-Svr-Logins.sql
  • This Script will act on the SQL Server Login named in each line by field linLoginName where linScope = Svr, linActiveNow = Yes or DryRun, linOpIVerb is not NULL.
  • Descriptions of Script function include syntax definitions showing how Table columns are used in the operation. The prefix lower-case characters in each name indicate the Table from which a column is sourced: rol for role definition table tblOpIRolesCreate, gr for Grant Permission definition table tblOpIRolesGRANT, fix for Fixed-Role definition table tblOpIRolesFIXED, lin for Login definition tblOpILogins, sch for tblOpISchemaCreate; usr for tblOpIUsers; and on for variable-definition table tblOpIOnObj.
  • linOpIVerb = DROP

    this Script will first drop all memberships of Login linLoginName in all Server Roles, and then DROP Server Login linLoginName.

    DROP Server Login

    -- DROP linLoginName Membership in foundRoles
    ALTER roleType -- from system lookup foundRole = SERVER
         ROLE foundRoleName -- from system lookup, linLoginName is member of foundRole
         DROP MEMBER linLoginName; -- drops membership of linLoginName from foundRole
        -- Loops through all memberships in found Roles.
    
    -- DROP Role membership
        DROP LOGIN linLoginName;
    
  • linRoleVerb = ADD/UPDATE

    If linLoginName does not exist it will be created. If Login linLoginName already exists when the Script is run with linRoleVerb = ADD/UPDATE, then Login linLoginName will be ALTERED.

    In each line, if linOpIVerb = ADD/UPDATE, this Script will CREATE or ALTER a Login to SQL Server using protocol and settings specified by field linOpILoginWindows for Windows, SQL Server, Certificate, Asymmetric Key, or Server Credential.

    Fields within tblOpILogins that apply to each Login protocol are specified in rows just above the Field Name header of that table. Use of the fields in SQL used to effectuate the action is also illustrated in images below in this section. A Field not used for a particular linLoginName will be ignored and may be left blank.

    For each line in tblOpILogins, this Script, linRoleVerb will additionally ADD, DROP or ignore Login linLoginName membership in Server Role linRoleName. Membership of existing linLoginName in multiple Server linRoleNames will be ADDED or DROPPED as specified in each of multiple lines in tblOpILogins.

    Finally, this Script will GRANT, REVOKE, or DENY individual Permissions outside of Role memberships, to individual Server Logins. For this, add a row for each such Permission in tblOpIRolesGRANT for each Server Login, specifying Permission variables. Specify grRole as the Server Login name linLoginName, and specify grRoleClass as "SvrLogin". This Script will execute each grVerb to GRANT, REVOKE, or DENY each specified Permission to each such specified Server Login.

    CREATE Server Login

    LoginCreate

    ALTER Server Login

    LoginAlter

    tblOpILoginsFieldValues
    tblOpILoginsFieldValues

  • References

    Login CREATE Reference

CREATE Database

This SSMS Script created from SQLServerRemote\ToolManagement.xlsm and residing in script directory as "CREATE_Server1DatabaseName1" creates a Program/Project database.

You must set variable @dbname, and the instance "SQLServerRemoteDB" before running the Script.

CREATE Database
/* SQL Server CREATE Database */

/* #VBTEXTJOIN: EXPORTWITH dbo; */

DECLARE @dbname nvarchar(128)
SET @dbname = N'SQLServerRemoteDB'

IF (NOT EXISTS (SELECT name FROM master.dbo.sysdatabases
  WHERE ('[' + name + ']' = @dbname OR name = @dbname)))
    BEGIN
      CREATE DATABASE SQLServerRemoteDB

    END;
GO

USE SQLServerRemoteDB;
GO

IF EXISTS(SELECT is_trustworthy_on, name
  FROM sys.databases
  WHERE name = 'SQLServerRemoteDB' AND is_trustworthy_on=0)
ALTER DATABASE SQLServerRemoteDB SET TRUSTWORTHY ON;

ALTER Database

DROP Database

DROP DATABASE [ IF EXISTS ] { database_name | database_snapshot_name } [ ,...n ] [;]

Database CONNECTION

To run Database SQL Scripts 3, 4, 5 you must first Connect the Script to the Database to which Script is directed.

If you're already connected to the Server, in SSMS open the <scriptname>.sql file. Then click in the open script tab and select the database that you want to execute on from the QUERY toolbar dropdown.

ConnectFromToolbar

If un-connected, or if you need to re-connect with a Login with permissions to execute the script, you can select the database to execute the open script on, from within the SSMS connection sequence. Open the <scriptname>.sql file, an right-click in it (red arrow starting in the middle of the figure below). Then follow the path shown to select "Connect...", "Options>>", "<Browse server...>", and select the database to exectue the script on.

ConnectionFromQueryCompositeEdit

You need to connect like this to manage database Schemas, User and Fixed Roles, and Database Users.

CREATE - ALTER - DROP Database Schema

CREATE Database Schema

  • tblOpISchemaCreate
    tblOpISchemaCreate
  • First: Connect to intended Database.
  • Using Script: SECURITY_3-CREATE-DB-Schema.sql
  • This Script will act on the Database Schema specified in each line by field schSchemaName where schScope = DB, schActiveNow = Yes or DryRun, schSchemaVerb is not NULL.
  • A Schema contain Tables and Views, and Database User Permissions can be set relative to Domain containing an object. So Tables with sensitive information could be held within a domain to which only specified Database Users have access. Such objects would be invisible and un-usable to Users without permissions on those Domains. The Permissions can be specified in a Database User Role; a User inherits its Permissions if assigned membership in that Role.
  • OpISQLServerSecurity puts Program data in tables in the dbo Domain, which is a SQL Server default Domain along with user and sys Domains. OpISQLServerSecurity default table values set up Domains, Logins and Database Users for Domains hr (Human Resources), fin (Finance), mgr (Management), and opi (OpI Security Tables). These domains could be used to contain Tables with project-related sensitive information.
  • When creating a Schema, objects contained by it can be created it the same CREATE command. This is not supported by the PowerOpI CREATE implementation; objects must be created in separate SSMS SQL commands and can be transferred to the SCHEMA by the ALTER opcode of this Script.
  • Descriptions of Script function include syntax definitions showing how Table columns are used in the operation. The prefix lower-case characters in each name indicate the Table from which a column is sourced: rol for role definition table tblOpIRolesCreate, gr for Grant Permission definition table tblOpIRolesGRANT, fix for Fixed-Role definition table tblOpIRolesFIXED, lin for Login definition tblOpILogins, sch for tblOpISchemaCreate; usr for tblOpIUsers; and on for variable-definition table tblOpIOnObj.
  • schSchemaIVerb = DROP

    This Script DROPs an existing Schema.

    DROP Schema

    DROP SCHEMA [IF EXISTS] schSchemaName;
    
  • schSchemaVerb = ADD/UPDATE

    If Database Schema schSchemaName does not exist it will be created.

    CREATE Database Schema
    CREATE SCHEMA schSchemaName AUTHORIZATION [schOwnerDomain\] schOwnerName;
        -- schOwnerName is typically db_securityadmin
    
  • schSchemaVerb = ALTER

    ALTER Database Schema

    Schema objects, such as a Table (dbo.tblTableName) are created within a Schema, for which the creator has ALTER SCHEMA permission within the Database in which the Object is created. That could be within the intended Schema, or in another Schema. An Object is created in a Schema different than its target Schema, can be transfered into the Target Schema, again by a creator with ALTER SCHEMA permission. This is the function of the ALTER opcode within Script SECURITY_3-CREATE-DB-Schema.sql.

    ALTER SCHEMA schSchemaName 
        TRANSFER [schTransferEntityType::][schTransferFromSchema.]schObjectToTransfer;
    
  • References

    SQL Server Security Overview

    Database Engine Permissions Poster (download PDF)

CREATE - ALTER - GRANT - DROP Database Role

CREATE Database Roles

  • tblOpIRolesCreate
    tblOpIRolesCreate

Database User GRANT-Role Permissions

  • tblOpIRolesGRANT
    tblOpIRolesGRANT
  • First: Connect to intended Database.
  • Using Script: SECURITY_4-CREATE-DB-GRANT-Roles.sql
  • This Script will act on the Database role named in each line by field rolRoleName where rolScope = DB, rolFixedGrant = Grant, rolActiveNow = Yes or DryRun, rolOpIVerb is not NULL.
  • This Script will act on Database-Level Permissions for each such rolRoleName where grScope = DB, grVerb is not NULL and rolRoleName = grRole.
  • This Script will act on Memberships by such rolRoleName into Database Role rolMemberOfRole where rolMemberOfVerb is not NULL.
  • Descriptions of Script function include syntax definitions showing how Table columns are used in the operation. The prefix lower-case characters in each name indicate the Table from which a column is sourced: rol for role definition table tblOpIRolesCreate, gr for Grant Permission definition table tblOpIRolesGRANT, fix for Fixed-Role definition table tblOpIRolesFIXED, lin for Login definition tblOpILogins, sch for tblOpISchemaCreate; usr for tblOpIUsers; and on for variable-definition table tblOpIOnObj.
  • rolOpIVerb = DROP

    For each Database role rolRoleName described by a line in tblOpIRolesCreate, OpISQLServerSecurity, Script first DROPs all memberships by that role, in other Database roles.

    Next, Script DROPs all memberships within the specified Database role.

    Finally, Script DROPs the specified Database role itself.

    -- DROP rolRolename Membership in otherRoles
    ALTER ROLE otherRoleName -- from system lookup, thisRole is member of otherRole
         DROP MEMBER rolRoleName; -- drops membership of thisRole in otherRole
        -- Loops through all memberships in other Roles.
    
    -- DROP members of rolRolName
    ALTER ROLE rolRoleName
        DROP MEMBER memberName; -- drops membership of memberName in thisRole
        -- Loops through all memberships within this Role.
    
    -- DROP Role
        DROP ROLE rolRoleName; -- roleDesc from system lookup of thisRole
    
  • rolOpIVerb = ADD/UPDATE

    Script First CREATEs a Database-Level Role, if it does not already exist. If a Database role by that name already exists, the existing role will be updated.

    CREATE ROLE rolRoleName AUTHORIZATION db_owner;
    

    For each Database role named by field rolRoleName described by a line in OpISQLServerSecurity tblOpIRolesCreate, the SSMS Script loops through permissions in tblOpIRolesGRANT, and executes the grVerb in each of those Database Role permissions to GRANT, DENY, or REVOKE each permission for that Role (where grRole = rolRoleName).

    Then Script executes rolMemberOfVerb to ADD or DROP membership of that Database Role rolRoleName in the Database role specified by rolMemberOfRole.

    Membership of rolRoleName in additional Database roles may be specified by adding a line for each ADD or DROP needed.

    Note use of Table column-name variables using lower-case prefixes that indicate the source table for the variable. Prefixes are enumerated above.

    -- Syntax definition
    GRANT <permission> [ ,...n ]
         [ ON [ <class_type> :: ] securable ]
         TO principal [ ,...n ]
              [ WITH GRANT OPTION ]
         [ AS <database_principal>
        ]
        [;]
    
    -- Implemented by Script using field specifications in tblOpIRolesCreate and tblOpIRolesGRANT
    grVerb grPermission
         [ grOnObject onObj [ onObjectName | rolDatabase /* if onObjectName=onDatabase */; ] ]
         TO grRole
              [ WITH GRANT OPTION -- if grVerb = GRANT and grGrantOpt = YES
                   | CASCADE -- if grVerb IN(REVOKE | DENY) and grGrantOpt = YES
              ] -- if grGrantOpt=Yes
         [ AS onObjectName | db_securityadmin /* if onObjectName=onDatabase */
         ] -- if grAS not NULL
    [ ; ]
    

    Finally, Script executes rolMemberOfVerb to add or drop membership of Database role rolRoleName in upstream Database role rolMemberOfRole.

    ALTER ROLE rolMemberOfRole rolMemberOfVerb MEMBER rolRoleName;
    

    Membership by a created Database role in additional upstream Database roles can be added or dropped by specifying rolMemberOfVerb = ADD or DROP in additional records/lines in tblOpIRolesCreate.

  • rolOpIVerb = ALTER

    No operation is defined for rolOpIVerb = ALTER. rolOpIVerb ADD/UPDATE uses CREATE and ALTER Role management keywords internally within its functionality, but much other functionality is embedded with this opcode so these SQL keywords are not offered directly by the OpISQLServerSecurity tool.

  • References

    CREATE Database Role

    GRANT Database Role Permissions

    ALTER Database Role

Database FIXED-Role Memberships

  • tblOpIRolesFIXED
    tblOpIRolesFIXED
  • First: Connect to intended Database.
  • Using Script: SECURITY_4-CREATE-DB-FIXED-Roles.sql
  • This Script will act on the Database FIXED role named by field rolRoleName in each line where rolScope = DB, rolFixedGrant = Fixed, rolActiveNow = Yes or DryRun, rolOpIVerb is not NULL.
  • This Script will act on Database Fixed Role memberships for rolRoleName where fixScope = DB (adding or dropping rolRoleName as a member of Database-level Fixed Role fixFixedRole to inherit or remove capabilities of fixFixedRole), and where fixVerb is not NULL and rolRoleName = fixRole.
  • This Script will act on Memberships in rolRoleName into Database Role rolMemberOfRole to inherit or remove rolMemberOfRole capabilities.
  • Descriptions of Script function include syntax definitions showing how Table columns are used in the operation. The prefix lower-case characters in each name indicate the Table from which a column is sourced: rol for role definition table tblOpIRolesCreate, gr for Grant Permission definition table tblOpIRolesGRANT, fix for Fixed-Role definition table tblOpIRolesFIXED, lin for Login definition tblOpILogins, sch for tblOpISchemaCreate; usr for tblOpIUsers; and on for variable-definition table tblOpIOnObj.
  • rolOpIVerb = DROP

    For each Database Fixed role rolRoleName described by a line in tblOpIRolesCreate, OpISQLServerSecurity Script first DROPs all memberships by that role, in other roles.

    Next, Script DROPs all memberships within the rolRoleName.

    Finally, Script DROPs Database Fixed role rolRoleName itself.

    -- DROP rolRoleName Membership in otherRoles
    ALTER ROLE otherRoleName -- from system lookup, thisRole is member of otherRole
         DROP MEMBER rolRoleName; -- drops membership of thisRole in otherRole
        -- Loops through all memberships in other Roles.
    
    -- DROP members of rolRoleName
    ALTER ROLE rolRoleName
        DROP MEMBER memberName; -- drops membership of memberName in thisRole
        -- Loops through all memberships within this Role.
    
    -- DROP Role
        DROP ROLE rolRoleName;
    
  • rolOpIVerb = ADD/UPDATE

    Script First CREATEs a Database Fixed Role by name rolRoleName, if it does not already exist. If a role by that name already exists, the existing role will be updated.

    CREATE ROLE rolRoleName AUTHORIZATION db_owner;
    

    For each Database Fixed role named by field rolRoleName by a line in tblOpIRolesCreate, OpISQLServerSecurity, Script loops through Database-level Fixed Roles in tblOpIRolesFIXED, and executes the fixVerb (ADD or DROP) in each of those Database FIXED Role permissions to ADD or DROP membership of fixRole (where fixRole = rolRoleName) in role fixFixedRole to inherit capabilities of fixFixedRole into fixRole (=rolRoleName), or drop those capabilities from fixRole (=roleRoleName).

    Note use of Table column-name variables using lower-case prefixes that indicate the source table for the variable. Prefixes are enumerated above.

    -- Implemented by Script using field specifications in tblOpIRolesCreate and tblOpIRolesFIXED
    ALTER ROLE fixFixedRole fixVerb MEMBER rolRoleName;
    

    Finally, Script executes rolMemberOfVerb opcode to add or drop membership of Database Fixed role rolRoleName in upstream Database Fixed role rolMemberOfRole.

    ALTER ROLE rolMemberOfRole rolMemberOfVerb MEMBER rolRoleName;
    

    Membership by a created Database Fixed role in additional upstream Database Fixed roles can be added or dropped by specifying rolMemberOfVerb = ADD or DROP in additional records/lines in tblOpIRolesCreate.

  • rolOpIVerb = ALTER

    No operation is defined for rolOpIVerb = ALTER. rolOpIVerb ADD/UPDATE uses CREATE and ALTER Role management keywords internally within its functionality, but much other functionality is embedded with this opcode so these SQL keywords are not offered directly by the OpISQLServerSecurity tool.

  • References

    Database Roles

    Database object identifiers

    Rules for T-SQL Identifiers

CREATE - ALTER - DROP Database User

The default table entries will set up default Database Users, associated with default Server Logins. Use them for setup and test.

Database Users described in the following descriptions and in the downloadable OpISQLServerTool, are named in usrUserName for the Role that they play: OpISuper, OpIManager, OpIDataOwner, OpIClient. Database User names match Server Login names. This is to show clearly the relationships among Logins, Database Users, Server Roles and Database User Roles. In production implementation, those Principals (Server Logins and Database Users) would be named for individuals, organizational positions, or the like; not by these default names.

Before you Go-Live, change these Principal names and passwords, or set up different Server Logins and Database Users for production use, and DISABLE these default Database Users. Every user of OpISQLServerSecurity will gain the same default user names and passwords on install or over time, and could possibly use them to access your database if you have left them intact - so disable the defaults before that happens.

CREATE Database Users

  • tblOpIUsers
    tblOpIUsers
  • First: Connect to intended Database.
  • Using Script: SECURITY_5-CREATE-DB-Users.sql
  • This Script will act on the Database User specified in each line by field usrUserName where usrScope = DB, usrActiveNow = Yes or DryRun, usrOpIVerb is not NULL.
  • Descriptions of Script function include syntax definitions showing how Table columns are used in the operation. The prefix lower-case characters in each name indicate the Table from which a column is sourced: rol for role definition table tblOpIRolesCreate, gr for Grant Permission definition table tblOpIRolesGRANT, fix for Fixed-Role definition table tblOpIRolesFIXED, lin for Login definition tblOpILogins, sch for tblOpISchemaCreate; usr for tblOpIUsers; and on for variable-definition table tblOpIOnObj.
  • usrOpIVerb = DROP

    this Script will first drop all memberships of Database User usrUserName in all Database User Roles, and then DROP Database User usrUserName itself.

    DROP User from Database

    -- DROP usrUserName Membership in foundRoles
    ALTER ROLE foundRoleName -- from system lookup, usrUserName is member of foundRole
         DROP MEMBER usrUserName; -- drops membership of usrUserName from foundRole
        -- Loops through all memberships in found Roles.
    
    -- DROP Role membership
        DROP USER IF EXISTS usrUserName;
    
  • usrRoleVerb = ADD/UPDATE

    If Database User usrUserName does not exist it will be created. If Database User usrUserName already exists when the Script is run with usrRoleVerb = ADD/UPDATE, then Database User usrUserName will be ALTERED.

    By nature of the ADD/UPDATE verb, the Database User usrUserName is associated with database usrDatabase, and optionally ADDED or DROPPED as specified by usrRoleVerb, as a member of Database User Role usrRoleName. Additional lines may be used to ADD/DROP the Database User usrUserName with additional Database Roles with database usrDatabase.

    In each line, fields usrIdentifierText and usrIdentifierToken accept free-form text to aid identification of the person associated with the Database User name usrUserName. That could include the person's full name, employee ID number, or other identifying information. Content of these fields remains within the OpISQLServerSecurity tool, and is not pushed into any database local or remote.

    -- CREATE the Database User usrUserName if not already exists
    CREATE USER [usrUserName] FROM LOGIN [usrUserName];
    
    -- Add the Database User as a member of specified role
    ALTER ROLE usrRoleName usrRoleVerb MEMBER usrUserName;
    
    -- GRANT permissions specific to Database User
    grVerb grPermission 
        [grOnObject onObj [onObjectName | rolDatabase /* if onObjectName = onDatabase*/; ] ]
        TO [usrUserName] 
            [WITH GRANT OPTION - if grVerb = GRANT and grGrantOpt = Yes
                | CASCADE -- if grVerb IN(REVOKE | DENY) and grGrantOpt = YES
            ] -- if grGrantOpt = Yes
        [AS onObjectName | usrDatabase /* if onObjectName = onDatabase */
        ] -- if grAS not NULL
    {;]
    

    Finally, this Script will GRANT, REVOKE, or DENY individual Permissions outside of Role memberships, to individual Database Users on each specified usrDatabase. For this, add a row for each such Permission in tblOpIRolesGRANT for each Database User, specifying Permission variables. Specify grRole as the Database User name usrUserName, and specify grRoleClass as "DBUser". This Script will execute each grVerb to GRANT, REVOKE, or DENY each specified Permission to each such specified Database User.

  • References

    Database User CREATE Reference

Push Tables to SQL Server

Introduction
There are many examples on this website that use Tables in Excel that are accessed and JOINed by a Database Environment to drive reports in Clients. This has worked well for projects in a tight management structure, even for pretty large programs across dispersed organizations.

For larger projects with more users who may be substantially distributed, a much more common practice is to use Tables that reside within databases in the Database Server environment. This provides better security, and better access by distributed remote users. Data integrity may be less dependent on networked links. This is the structure supported and advocated by IT organizations - although this leads to a big hole in tools available for Program Management as I will describe shortly!

  • PowerOpI Tools Use Excel as a Front-End
    The PowerOpI tools have used Excel as data-owner front-end, providing source data.
  • Why Excel?
    Here's why: I've participated in numerous Program Management environments, developing system products and services that involve hardware as well as software. There are no Program Management tools that support logistics of large hardware development efforts, let alone hybrid mixed-technology products. There are tools for software project management for backlog capture, refinement, and communication; Jira or Monday for example. Beyond that, Program Management tools for larger project, especially for hardware, are a spreadsheet, a presentation tool, and maybe a scheduling tool depending on PM methodology tied to organization. So Microsoft Excel, Word, Powerpoint and Project; or Google Sheets, Docs, Slides, and Smartsheet or a scheduler embedded in the backlog manager. Or Jira, Jamboard, and Smartsheet (maybe).
  • No Common Tools So PMs Develop Local Tools But Few PMs know VBA, SQL, M
    Nevertheless, programs including hardware involve complex logistics involving interdependencies of function and timing, materials, integration and test. Even programs including complex parallel software development efforts can have dependencies, schedule, and logistics of underlying systems and suppliers. There really aren't tools, good bad or indifferent, aimed at these problems. Instead, PMs usually use Excel for these problems, using varying involvement of basic cell logic and functions, VLOOKUPs, and Pivot Tables. A few Program Managers use VBA to extend functionality. But more sophisticated tools including Databases or PowerQuery for JOIN require SQL or M and more topics outlined on this website. That is extremely rare among Program Managers and host organizations.
  • No Internal Programming Support for Program Management
    Furthermore, the tool problem is not even recognized in Program Management culture, and support from other organizations for IT and custom solutions is unknown... And yet, note the above advocacy of tool structure! I've never seen any organizational support providing Program Management tools even once, beyond basic PC maintenance. Where support exists, it focuses on limited-access function within in-house databases (BOM, Purchasing, HR) that are inevitably driven by revenue-generation (manufacturing and marketing), personnel (HR), and formal accounting (finance/CFO).
  • Given ties of those databases to sensitive corporate operation, even database tools that might be used by PM for development projects involve bureaucratic, time-consuming permissions and processes that don't support development programs. Access to these systems by Program Managers for efforts that may last a year or two and which would include content that will never itself become revenue product (systems and components for integration, destructive test; NRE for development; project staff planning) is just not a consideration by organizations driving those systems.
  • Logistics Are Often Custom by Technology, Product, and Company
    To make matters worse, technologies and organizational structures make most solutions developed locally by Program Managers to be custom-built by local PMs. Varying products with different fab, build, test, integration processes require varying logistics. Corporate specifics of purchasing, expense, and capitalization vary and often don't consider timing of development projects.
  • Fiscal year specifics vary from company to company, lining up on different months and maybe not recognizing common calendar month boundaries. PM tools have to talk the languages of both Fiscal and calendar dates depending on audience. Program finance reporting is by fiscal year whereas logistics with manufacturing operation and supply chain are based on calendar dates. Logistics of expense and capital handling of larger prototypes; approvals for PO placement with material lead-time many months in advance; capitalization vs. expense for environmental test systems may not be well defined.
  • So tools for managing these programs are invariably custom, developed in-house by the Program Managers themselves and limited by available individual knowledge and skill set; and because of that rarely even common among a given set of Program Managers. Also, probably because of these broad variances, there are no commercial tools addressing these logistics management requirements.
  • So this is the hole.
  • Database Tools Trump Spreadsheets for Complex Projects
    Nevertheless, use of a Database Environment with Excel has proven extremely valuable, supporting planning and currency of projected of "best path from here", helping decision support, coordinating dependencies and processes across organizations, supporting problem identification and resolution, expense/capital/revenue planning; and execution across the board. Update and currency of these database-tool-based plans is more timely, because of substantially reduced effort than basic spreadsheet-based solutions - if such updates are even possible for spreadsheet tools.
  • The cost of this is some additional skills in a limited set of PMs in an organization. The structure of PowerOpI tools greatly limits that complexity, making such coordination and projection much more available, and supporting implementation of standard constructs and customizations within tools already well-known by Program Managers.
  • Most Program Managers Know Excel
    An extremely common attribute among Program Managers is that most of them know Excel or Spreadsheets to some degree. So using that, PowerOpI puts source data managing even complex projects into Excel, and provides common basic QUERYs that pull data in from Excel, execute the JOIN, and provide output for Pivot Tables and standard reporting back in Excel. Where some SQL would help in customizing or extending QUERY or JOIN, adding a Table or a new Table Field, support is provided in Excel, requiring fairly minimal training of a smart PM to accomplish that capability.
  • There is still an un-resolved limitation - some use of SSMS is needed for some Login and Database User Roles. We are not able to execute some SQL from Excel, notably to create a database, a stored procedure, create or update a Table. This could be some capability yet to be figured out, or a limitation of OLEDB as used by Excel that could be resolved by using ODBC (more on that to follow). The current solution to these database-structural issues is to export scripts that can be executed from SSMS. These Scripts at least are configured appropriately to reflect the database targeted.
  • PowerOpI Tools Leverage PM Excel Knowledge to Provide Database Tools for PM
    Okay, having said all that, larger programs still benefit from Tables residing within the Database Environment.
  • So the solution from PowerOpI is to keep source data within Excel, managed among a set of Data Owners, and to provide a mechanism to push that source data from Excel up into database-resident tables. This leverages Excel tool knowledge that data-owner PMs already have. This supports customizations; retains synchronization of data among source tables; and improves time to update with concurrent Client use. Furthermore, security is enhanced by enabling use of database Views, with consequent reduction to scope of database permissions required for Client users, with no reduction of information available even for varying Client types.

So here's how Table Push works in PowerOpI Tools, to push data from Excel Tables up into a Program Database in SQL Server.

Table Column Datatype

  • In Excel ToolSource.xlsm
  • Before an Excel Table can be pushed to SQL Server, a SQL Server Database Table must be created that matches datatypes of each column within the Excel Table to be pushed.
  • Excel is so good at coercing datatype for data that may be used in many ways, that Excel developers rarely even think about it. However, datatype of a value in Excel may be quite different than its formatted display. Furthermore, means to determine actual datatype of a value in Excel are somewhat limited.
  • SQL Server is somewhat stricter about datatype. When pushing data of unspecified datatype to a database Table, SQL Server attempts to determine its datatype and may be right or wrong in deciding on a datatype consistent with use of the value as you have intended. As a common example, datatype of an Excel value you use as a number that is pushed to SQL Server, may be determined by SQL Server as a character type like VARCHAR, for a value you want to represent as an INT or FLOAT for a calculation in SQL. Then arithmetic won't work for TotalQuantity = AssyQuantity * ItemQuantityPerAssy in a QUERY returning a Table driving an Excel Pivot Table.
  • PowerOpI tools address this by containing a Table called "qGETDatatypes" on tab DBdatatypes. This is contained in Toolsource.xlsm files containing data tables.
  • tblGETDatatypes
    PushTablesDataTypes
  • The Table includes a row for each field in each Excel Table, stating its datatype to be established in a SQL Table. Each field named in ColumnName specifies a SQL DataType, and associated parameters used in SQL Table CREATION.
    CREATE TABLE parameters
    SQL Datatypes
  • This Excel Table is referenced when CREATING a SQL Table, and when data values are INSERTED into it. The Excel Table default illustrates settings for most common datatypes used by PowerOpI tools, for reference when creating new Table columns. SQL Server CREATE TABLE provides a huge set of options. PowerOpI Tools creates simple Tables, only creating columns with FieldNames and specifying the datatype of each. Tables created by other means and incorporating more options may still be read by PowerOpI tools for use in QUERYs, but PowerOpI cannot push into such Tables.
  • When an Excel designer adds a column to a table, there should also be a row added to table qGETDatatypes specifying its datatype. When the Table is updated to include a new column, the datatype specified in qGETDatatypes is used if available; otherwise the PowerOpI tool attempts to determine its datatype from Excel information; and if not available by those means, SQL Server attempts to determine the datatype and often chooses VARCHAR.
  • Just to save you some debug time: Note that character strings coming back to Excel from a QUERY on SQL Server, from a column of type nvarchar(max),under at least some conditions, come back as null strings. A known error case occurs when accessing nvarchar(max) strings from SQL Server to Excel via ODBC. To be safe, when creating a Table in SQL Server, instead define datatype for wide character string columns as something like nvarchar(250).
  • When all is said and done, if a datatype is determined incorrectly for a particular use in a QUERY of SQL Server, the QUERY can always CAST the datatype to what is needed by the QUERY. That works, but imposes on the QUERY designer identification of the problem, and the solution.

Executing TABLE PUSH operations: CREATE, INSERT, DROP, and Skip

  • tblPushTables
  • PushTablesControl
  • Using Excel tblPushTables
  • On push of button Push Tables, this Script will act on the Excel Table specified in each line by fields puWkTable and puSchema and database name from Excel workbook field gblServerDatabaseName1 at the bottom of its DataSources tab, executing action specified by puVerb affecting SQL Server Database Table puDBTable.
  • puVerb = Skip

    On push of button Push Tables, no action will be taken by this line and no Database Table will be affected.

  • puVerb = Create table

    On push of button Push Tables, a <scriptname>.sql Script file will be created named TablesPush-Directive.sql in directory specified by Excel Variables tab variable ExportTo, to be executed from SSMS. That Script will CREATE a Database Table for each row in tblPushTables with puVerb = Create Table. In the Script, database name is taken from Excel workbook field gblServerDatabaseName1 at the bottom of its DataSources tab.

    Note how Table name, field names, values and order match tblGETDataTypes illustrated above.

    Each DB Table will be named as specified by column puDBTable, with DB Table columns created and named as columns in puWkTable and with each DB Table column datatype as specified in tblGETDatatypes. If no datatype is specified, Excel and SQL Server will attempt to determine suitable type from the data itself, but that may be inaccurate.

    The Database Table will be created with Columns, Column Names, and Column datatypes, but no data will be pushed up from Excel Table puWkTable by Create table. It is anticipated that Create table only needs to be executed occasionally when the DB Table is created, and when its columns are added, deleted or otherwise changed. Data within the DB Table columns is be pushed up to such an existing DB Table by subsequently executing puVerb = Insert values.

    The following illustrates a Table creation script, created automatically:

    DROP TABLE IF EXISTS [SQLServerRemoteDB].[dbo].[xlAlloc]; 
     
    SET ANSI_NULLS ON 
    SET QUOTED_IDENTIFIER ON 
    CREATE TABLE [SQLServerRemoteDB].[dbo].[xlAlloc] ( 
    [akAllocLine] INT NULL,     -- converted from Excel General 
    [akOrg] NVARCHAR(max) NULL,     -- converted from Excel General 
    [akTest] NVARCHAR(max) NULL,     -- converted from Excel General 
    [akSysID] NVARCHAR(max) NULL,     -- converted from Excel General 
    [akSysRev] NVARCHAR(max) NULL,     -- converted from Excel General 
    [akElement] NVARCHAR(max) NULL,     -- converted from Excel General 
    [akRev] NVARCHAR(max) NULL,     -- converted from Excel General 
    [akQty] INT NULL,     -- converted from Excel General 
    [akPurchReq] NVARCHAR(max) NULL,     -- converted from Excel General 
    [akAllocRevKey] NVARCHAR(max) NULL,     -- converted from Excel General 
    [akAllocItemKey] NVARCHAR(max) NULL,     -- converted from Excel General 
    [akEnetStructKey] NVARCHAR(max) NULL,     -- converted from Excel General 
    [akNotes] NVARCHAR(max) NULL     -- converted from Excel General 
    ) 
    ON [PRIMARY]; 
     
    RETURN; 
    

    When executing this Script in SSMS: First Connect to intended Database.

  • puVerb = Insert Values

    On push of button Push Tables, a <scriptname>.sql Script file will be created named TablesPush-Directive.sql in directory specified by Excel Variables tab variable ExportTo, to be executed from SSMS. In the Script, database name is taken from Excel workbook field gblServerDatabaseName1 at the bottom of its DataSources tab.

    That Script will, for each row in tblPushTables with puVerb = Insert Values, first delete existing rows in Database Table puDBTable, then INSERT VALUES from all data rows from Excel Table puWkTable, with any changed columns, into Database Table puDBTable.

    Note how Table name, field names, values and order match tblGETDataTypes illustrated above.

    The following illustrates an Insert Values script created automatically:

    DELETE FROM [SQLServerRemoteDB].[dbo].[xlAlloc]; 
     
    INSERT INTO [SQLServerRemoteDB].[dbo].[xlAlloc] 
    ([akAllocLine], [akOrg], [akTest], [akSysID], [akSysRev], [akElement], [akRev], [akQty],
        [akPurchReq], [akAllocRevKey], [akAllocItemKey], [akEnetStructKey], [akNotes]) 
    VALUES  
    (9, N'HW', N'Bringup', N'HW-1', N'P0-B', N'StorageAppXM', N'P0-B', 1,
        N'11489', N'StorageAppXM:P0-B', N'StorageAppXM', N'HW-1:StorageAppXM', N'Direct allocation'),  
    (10, N'HW', N'Bringup', N'HW-1', N'P0-B', N'Europa', N'P0-B', 2, 
        N'11837', N'Europa:P0-B', N'Europa', N'HW-1:Europa', N'Direct allocation');
    -- ... with one line comma-separated, for each following Table row,
    --      and note semicolon; on last inserted row.
    RETURN; 
    
    

    When executing this Script in SSMS: First Connect to intended Database.

  • puVerb = Drop Table

    On push of button Push Tables, a <scriptname>.sql Script file will be created named TablesPush-Directive.sql in directory specified by Variables tab variable ExportTo, to be executed from SSMS. That Script will DROP the existing Database Table puDBTable for each row in tblPushTables with puVerb = Drop Table.

    The following illustrates Drop Tables script created automatically:

    DROP TABLE IF EXISTS [SQLServerRemoteDB].[dbo].[xlAlloc]; 
     
    RETURN; 
    

    When executing this Script in SSMS: First Connect to intended Database.

Use Views for Improved Client Security

  • A VIEW is a virtual table in a SQL Server Database, that can be QUERied from an Excel or other client to drive Client Pivot Tables or other reporting structures. The VIEW is created by a Data Owner and contains an embedded QUERY determining JOINS, fields, and calculations to expose to a Client QUERY of the VIEW.
  • The embedded QUERY structure is hidden from Client, so no Database structure is exposed to Client. Furthermore, the Client does not need to execute a Stored Procedure on the server to return the VIEW content, so Execution and View Permissions for SPs on the Database are not required, thus removing exposure of the Database to execution of unintended SPs.
  • CREATE VIEW
  • Create a VIEW in SQLServerRemote\ToolSource.xlsm; requires Server Login with OpIDataOwner Role:

    /* CREATE vwProtoTables */
    USE SQLServerRemoteDB;
    GO
    /* #VBTEXTJOIN: EXPORTWITH dbo; */
    
    SET ANSI_NULLS ON
    GO
    SET QUOTED_IDENTIFIER ON
    GO
    
    /* Create View local to [COBALT\SQLEXPRESS].[SQLServerRemoteDB], to be used from Client.
    View pulls data from [COBALT\SQLEXPRESS].[SQLServerRemoteDB] tables */
    CREATE OR ALTER VIEW [dbo].[vwProtoTables]
    AS
      SELECT [dbo].[xlAlloc].*, [dbo].[xlBOM].*, [dbo].[xlDates].*, [dbo].[xlBuilds].*, 
        [dbo].[xlElements].*, [dbo].[xlPR].*, 
        [dbo].[xlAlloc].[akQty] * [dbo].[xlBOM].[bmSubQty] AS QtyExt, 
        [dbo].[xlAlloc].[akQty] * [dbo].[xlBOM].[bmSubQty] * [dbo].[xlBOM].[bmSubCost] AS CostExt
      FROM [dbo].[xlAlloc]
      LEFT OUTER JOIN [dbo].[xlBOM] 
        ON [dbo].[xlAlloc].[akAllocRevKey] = [dbo].[xlBOM].[bmBOMItemKeyIn]
      LEFT OUTER JOIN [dbo].[xlDates] 
        ON [dbo].[xlBOM].[bmBOMItemKeyOut] = [dbo].[xlDates].[daSchedItemRevKey]
      LEFT OUTER JOIN [dbo].[xlBuilds] 
        ON [dbo].[xlDates].[daSchedItemRevKey] = [dbo].[xlBuilds].[bdBuildRevKey]
      LEFT OUTER JOIN [dbo].[xlElements] 
        ON [dbo].[xlBOM].[bmBOMItemKey] = [dbo].[xlElements].[emElementItemKey]
      LEFT OUTER JOIN [dbo].[xlPR] 
        ON CAST([dbo].[xlAlloc].[akPurchReq] AS VARCHAR) = CAST([dbo].[xlPR].[prPurchReq] AS VARCHAR)
    
  • QUERY a VIEW On any Client ToolClientRemote.xlsm using a SQL QUERY to bring back the VIEW Table to Excel to drive a Pivot Table or other report. Requires server Login with OpIClient role:

    /* QUERY vwProtoTables */
    SET ANSI_NULLS ON
    SET QUOTED_IDENTIFIER ON
    
    SELECT vwTables.*
    FROM [SQLServerRemoteDB].[dbo].[vwProtoTables] AS vwTables;