Make Funcparser support non-string returns; more tests

CHANGELOG.md

@@ -40,6 +40,8 @@
   code style and paradigms instead of relying on `Scripts` for everything. 
 - Expand `CommandTest` with ability to check multipler msg-receivers; inspired by PR by 
   user davewiththenicehat. Also add new doc string.
+- Add central `FuncParser` as a much more powerful replacement for the old `parse_inlinefunc` 
+  function.
 
 ### Evennia 0.9.5 (2019-2020)
 

docs/source/Components/FuncParser.md

@@ -2,39 +2,58 @@
 
 The [FuncParser](api:evennia.utils.funcparser.FuncParser) extracts and executes 'inline functions'
 embedded in a string on the form `$funcname(args, kwargs)`. Under the hood, this will 
-lead to a call to a same-named Python function you control. The inline function call will be 
+lead to a call to a Python function you control. The inline function call will be replaced by 
-replaced by the return from the function.
+the return from the function.
 
-A common use is to grant common players the ability to create dynamic content without access to 
+```python 
-Python. But inline functions are also potentially useful for developers. 
+from evennia.utils.funcparser import FuncParser
 
-Here are some examples:
+def _square(*args, **kwargs):
+    """This will be callable as $square(number) in string"""
+    return float(args[0]) ** 2
+
+parser = FuncParser({"square": _square})
+
+parser.parse("We have that 4 x 4 is $square(4).")
+"We have that 4 x 4 is 16."
+
+```
+
+Normally the return is always converted to a string but you can also retrieve other data types
+from the function calls:
+
+```python
+parser.parse_to_any("$square(4)")
+16
+```
+
+To show a `$func()` verbatim in your code without parsing it, escape it as either `$$func()` or `\$func()`.
+
+The point of inline-parsed functions is that they allow users to call dynamic code without giving 
+regular users full access to Python. You can supply any python function to process the users' input. 
+
+Here are some more examples: 
 
     "Let's meet at our guild hall. Here's how you get here: $route(Warrior's Guild)."
 
-In this example, the `$route()` call would be evaluated as an inline function call. Assuming the game
+This can be parsed when sending messages, the users's current session passed into the callable. Assuming the 
-used a grid system and some path-finding mechanism, this would calculate the route to the guild
+game used a grid system and some path-finding mechanism, this would calculate the route to the guild 
 individually for each recipient, such as:
 
-    "Let's meet at our guild hall. Here's how you get here: north,west,north,north.
+    player1: "Let's meet at our guild hall. Here's how you get here: north,west,north,north.
-    "Let's meet at our guild hall. Here's how you get here: south,east.
+    player2: "Let's meet at our guild hall. Here's how you get here: south,east.
-    "Let's meet at our guild hall. Here's how you get here: south,south,south,east.
+    player3: "Let's meet at our guild hall. Here's how you get here: south,south,south,east.
 
 It can be used (by user or developer) to implement _Actor stance emoting_ (2nd person) so people see
-different variations depending on who they are (the [RPSystem contrib](../Contribs/Contrib-Overview) does this in 
+different variations depending on who they are (the [RPSystem contrib](../Contribs/Contrib-Overview) does 
-a different way for _Director stance_):
+this in a different way for _Director stance_):
 
     sendstr = "$me() $inflect(look) at the $obj(garden)."
     
-    I see: "You look at the Splendid green Garden."
+    I see: "You look at the Splendid Green Garden."
-    others see: "Anna looks at the Splendid green Garden."
+    others see: "Anna looks at the Splendid Green Garden."
 
-One could do simple mathematical operations ...
+... embedded dice rolls ...
-   
-    "There are $eval(4**2) possibilities ..."
-    "There are 16 possibilities ..."    
-    
-... Or why not embedded dice rolls ...
 
     "I make a sweeping attack and roll $roll(2d6)!"
     "I make a sweeping attack and roll 8 (3+5 on 2d6)!"
@@ -44,6 +63,7 @@ Function calls can also be nested. Here's an example of inline formatting
     "This is a $fmt('-' * 20, $clr(r, red text)!, '-' * 20")
     "This is a --------------------red text!--------------------" 
 
+
 ```important::
     The inline-function parser is not intended as a 'softcode' programming language. It does not
     have things like loops and conditionals, for example. While you could in principle extend it to 
@@ -51,24 +71,18 @@ Function calls can also be nested. Here's an example of inline formatting
     Evennia expects you to do in a proper text editor, outside of the game, not from inside it.
 ```
 
-
 ## Standard uses of parser
 Out of the box, Evennia applies the parser in two situations: 
 
-### Inlinefuncs 
+### Inlinefunc parsing
 
-The original use for inline function parsing. When enabled (disabled by default), Evennia will 
+This is inactive by default. When active, Evennia will run the parser on _every outgoing string_
-apply the parser to every client-bound outgoing message. This is per-Session and 
+from a character, making the current [Session](./Sessions) available to every callable. This allows for a single string 
-`session=<current_session>` is always passed into each callable. This allows for things like 
+to appear differently to different users (see the example of `$route()` or `$me()`) above). 
-the per-receiver `$route` in the example above.
 
-- To enable inlinefunc parsing, set `INLINEFUNC_ENABLED=True` in your settings file 
+To turn on this parsing, set `INLINEFUNC_ENABLED=True` in your settings file. You can add more callables in
-  (`mygame/server/conf/settings.py`) and reload.
+`mygame/server/conf/inlinefuncs.py` and expand the list `INLINEFUNC_MODULES` with paths to modules containing
-- To add more functions, you can just add them to the pre-made module in 
+callables.
-  `mygame/server/conf/inlinefuncs.py`. Evennia will look here and use all top-level functions you add
-  (unless their name starts with an underscore).
-- If you want to get inlinefuncs from other places, `INLINEFUNC_MODULES` is a list of the paths
-  Evennia will use to find new modules with callables. See defaults in `evennia/settings_default.py`.
   
 These are some example callables distributed with Evennia for inlinefunc-use.
 
@@ -85,110 +99,174 @@ These are some example callables distributed with Evennia for inlinefunc-use.
   without the preceeding `|`. If no endcolor is given, the string will go back to neutral. 
   so `$clr(r, Hello)` is equivalent to `|rHello|n`. 
 
-
 ### Protfuncs
 
 Evennia applies the parser on the keys and values of [Prototypes definitions](./Prototypes). This 
-is meant for a user of the OLC to create prototypes with dynamic content (such as random stats). 
+is mainly used for in-game protoype building. The prototype keys/values are parsed with the 
+`FuncParser.parser_to_any` method so the user can set non-strings on prototype keys.
 
 See the prototype documentation for which protfuncs are available.
     
-    
 ## Using the FuncParser 
     
 You can apply inline function parsing to any string. The 
 [FuncParser](api:evennia.utils.funcparser.FuncParser) is found in `evennia.utils.funcparser.py`. 
-Here's how it's used:
 
 ```python
 from evennia.utils import funcparser
 
 parser = FuncParser(callables, **default_kwargs)
-parsed_string = parser.parser(input_string, raise_errors=False, **reserved_kwargs)
+parsed_string = parser.parser(input_string, raise_errors=False, 
+                              escape=False, strip=False, 
+                              return_str=True, **reserved_kwargs)
 
+# callables can also be passed as paths to modules
+parser = FuncParser(["game.myfuncparser_callables", "game.more_funcparser_callables"])
 ```
 
-The `callables` is either a `dict` mapping `{"funcname": callable, ...}`, a python path to 
+- `callables` - This is either a `dict` mapping `{"funcname": callable, ...}`, a python path to 
 a module or a list of such paths. If one or more paths, all top-level callables (whose name 
 does not start with an underscore) in that module are used to build the mapping automatically.
-
+- `raise_errors` - By default, any errors from a callable will be quietly ignored and the result 
-By default, any errors from a callable will be quietly ignored and the result will be that 
+  will be that the failing function call will show as if it was escaped. If `raise_errors` is set, 
-the un-parsed form of the callable shows in the string instead. If `raise_errors` is set, 
+  then parsing will stop and the error raised. It'd be up to you to handle this properly.
-then an error will stop parsing and a `evennia.utils.funcparser.ParsingError` will be raised
+- `escape` - Returns a string where every `$func(...)` has been escaped as `\$func()`. This makes the 
-with a string of info about the problem. It'd be up to you to handle this properly.
+  string safe from further parsing.
-
+- `strip` - Remove all `$func(...)` calls from string (as if each returned `''`).
-The default/reserved keywords are optional and allow you to pass custom data into _every_ function
+- `return_str` - When `True` (default), `parser` always returns a string. If `False`, it may return 
-call. This is great for including things like the current session or config options. See the next
+  the return value of a single function call in the string. This is the same as using the `.parse_to_any` 
-section for details. 
+  method.
-
+- The `**default/reserved_keywords` are optional and allow you to pass custom data into _every_ function
+  call. This is great for including things like the current session or config options. Defaults can be
+  replaced if the user gives the same-named kwarg in the string's function call. Reserved kwargs 
+  are always passed, ignoring defaults or what the user passed.
 
 ```python
-parser = funcparser.FuncParser(callables, test='bar')
+
-result = parser.parse("$header(foo)")
+def _test(*args, **kwargs):
+    # do stuff
+    return something
+
+parser = funcparser.FuncParser({"test": _test}, mydefault=2)
+result = parser.parse("$test(foo, bar=4)", myreserved=[1, 2, 3])
+
 ```
 
-Here the callable (`_header` from the first example) will be called as `_header('foo', test='bar')`. All
+Here the callable will be called as `_test('foo', bar='4', mydefault=2, myreserved=[1, 2, 3])`. 
-callables called through this parser will get this extra keyword passed to them. These does _not_ have 
+Note that everything given in the `$test(...)` call will enter the callable as strings. The 
-to be strings.
+kwargs passed outside will be passed as whatever type they were given as. The `mydef` kwarg
+could be overwritten by `$test(mydefault=...)` but `myreserved` will always be sent as-is, ignoring
+any same-named kwarg given to `$test`.
 
-Default keywords will be overridden if changed in the function call: 
+## Defining custom callables
-
-```python
-result = parser.parse("$header(foo, test=moo)")
-```
-
-Now the callable will be called as `_header('foo', test='moo'`) instead. Note that the values passed
-in from the string will always enter the callable as strings.
-
-If you want to _guarantee_ a certain keyword is always passed, you should pass it when you call `.parse`:
-
-``` python
-result = parser.parser("$header(foo, test=moo)", test='override')
-```
-
-The kwarg passed with `.parse` overrides the others, so now `_header('foo', test='override')` will
-be called. Like for default kwargs, these keywords do _not_ have to be strings. This is very useful 
-when you must pass something for the functionality to work. You may for example want to pass the 
-current user's Session as `session=session` so you can customize the response per-user.
-
-
-## Callables
 
 All callables made available to the parser must have the following signature:
 
 ```python
 def funcname(*args, **kwargs):
     # ...
-    return string
+    return something
 ```
 
-It's up to you as the dev to correctly parse all possible input. Remember that this may be called 
+As said, the input from the top-level string call will always be a string. However, if you 
-by untrusted users. If the return is not a string, it will be converted to one, so make sure this
+nest functions the input may be the return from _another_ callable. This may not be a string.
-is possible. 
+Since you should expect users to mix and match function calls, you must make sure your callables 
+gracefully can handle any input type.
 
-> Note, returning nothing is the same as returning `None` in Python, and this will convert to a 
+On error, return an empty/default value or raise `evennia.utils.funcparser.ParsingError` to completely 
-> string `"None"`. You usually want to return the empty string `''` instead.
+stop the parsing at any nesting depth (the `raise_errors` boolean will determine what happens).
 
-While the default/reserved kwargs can be any data type, the data from the parsed function call 
+Any type can be returned from the callable, but if its embedded in a longer string (or parsed without 
-itself will always be of type `str`. If you want more complex operations you need to convert 
+`return_str=True`), the final outcome will always be a string.
-from the string to the data type you want.
 
-Evennia comes with the [simpleeval](https://pypi.org/project/simpleeval/) package, which
+First, here are two useful tools for converting strings to other Python types in a safe way:
-can be used for safe evaluation of simple (and thus safe) expressions.
+
+- [ast.literal_eval](https://docs.python.org/3.8/library/ast.html#ast.literal_eval) is an in-built Python
+  function. It
+  _only_ supports strings, bytes, numbers, tuples, lists, dicts, sets, booleans and `None`. That's
+  it - no arithmetic or modifications of data is allowed. This is good for converting individual values and
+  lists/dicts from the input line to real Python objects.
+- [simpleeval](https://pypi.org/project/simpleeval/) is imported by Evennia. This allows for safe evaluation
+  of simple (and thus safe) expressions. One can operate on numbers and strings with +-/* as well 
+  as do simple comparisons like `4 > 3` and more. It does _not_ accept more complex containers like 
+  lists/dicts etc, so the two are complementary to each other.
+
+First we try `literal_eval`. This also illustrates how input types work.
+
+```python
+from ast import literal_eval
+
+def _literal(*args, **kwargs):
+    if args:
+        try:
+            return literal_eval(args[0])
+        except ValueError:
+            pass
+    return ''
+   
+def _add(*args, **kwargs):
+  if len(args) > 1:
+      return args[0] + args[1]
+  return ''
+
+parser = FuncParser({"literal": _literal, "add": _add})
+```
+
+We first try to add two numbers together straight up
+
+```python
+parser.parse("$add(5, 10)")
+"510"
+```
+The result is that we concatenated the strings "5" + "10" which is not what we wanted. This 
+because the arguments from the top level string always enter the callable as strings. We next
+try to convert each input value:
+
+```python
+parser.parse("$add($lit(5), $lit(10))")
+"15"
+parser.parse_to_any("$add($lit(5), $lit(10))")
+15
+parser.parse_to_any("$add($lit(5), $lit(10)) and extra text")
+"15 and extra text"
+```
+Now we correctly convert the strings to numbers and add them together. The result is still a string unless
+we use `parse_to_any` (or `.parse(..., return_str=False)`). If we include the call as part of a bigger string,
+the outcome is always be a string. 
+
+In this case, `simple_eval` makes things easier:
+
+```python
+from simpleeval import simple_eval
+
+def _eval((*args, **kwargs):
+    if args:
+        try:
+            return simple_eval(args[0])
+        except Exception as err:
+            return f"<Error: {err}>"
+            
+parser = FuncParser({"eval": _eval})
+parser.parse_to_any("5 + 10")
+10
+
+```
+
+This is a lot more natural in this case, but `literal_eval` can convert things like lists/dicts that the
+`simple_eval` cannot. Here we also tried out a different way to handle errors - by letting an error replace
+the `$func`-call directly in the string. This is not always suitable.
 
 ```warning::
-   Inline-func parsing can be made to operate on any string, including strings from regular users. It may 
+   It may be tempting to run Python's in-built `eval()` or `exec()` commands on the input in order to convert 
-   be tempting to run the Python full `eval()` or `exec()` commands on the input in order to convert it 
+   it from a string to regular Python objects. NEVER DO THIS. The parser is intended for untrusted users (if
-   from a string to regular Python objects. NEVER DO THIS. This would be a major security problem since it 
+   you were trusted you'd have access to Python already). Letting untrusted users pass strings to eval/exec 
-   would allow the user to effectively run arbitrary Python code on your server. There are plenty of 
+   is a MAJOR security risk. It allows the caller to effectively run arbitrary Python code on your server. 
-   examples to find online showing how a malicious user could mess up your system this way. If you ever 
+   This is the way to maliciously deleted hard drives. Just don't do it and sleep better at night.
-   decide to use eval/exec you should be 100% sure that it operates on strings that untrusted users 
-   can't modify.
 ```
 
-## Example of usage
+## Example:
 
-Here's a simple example 
+An 
 
 ```python
 from evennia.utils import funcparser
@@ -223,4 +301,3 @@ We nest the functions so the parsed result of the above would be something like
 This is the current uptime:
 ------- 343 seconds ------- 
 ```
-

evennia/utils/funcparser.py

@@ -118,7 +118,7 @@ class FuncParser:
             escape_char (str, optional): Prepend characters with this to have
                 them not count as a function. Default is `\\`.
             max_nesting (int, optional): How many levels of nested function calls
-                are allowed, to avoid exploitation.
+                are allowed, to avoid exploitation. Default is 20.
             **default_kwargs: These kwargs will be passed into all callables. These
                 kwargs can be overridden both by kwargs passed direcetly to `.parse` _and_
                 by kwargs given directly in the string `$funcname` call. They are
@@ -216,13 +216,18 @@ class FuncParser:
 
         try:
             return func(*args, **kwargs)
+        except ParsingError:
+            if raise_errors:
+                raise
+            return str(parsedfunc)
         except Exception:
             logger.log_trace()
             if raise_errors:
                 raise
             return str(parsedfunc)
 
-    def parse(self, string, raise_errors=False, escape=False, strip=False, **reserved_kwargs):
+    def parse(self, string, raise_errors=False, escape=False,
+              strip=False, return_str=True, **reserved_kwargs):
         """
         Use parser to parse a string that may or may not have `$funcname(*args, **kwargs)`
         - style tokens in it. Only the callables used to initiate the parser
@@ -238,6 +243,9 @@ class FuncParser:
                 are not executed by later parsing.
             strip (bool, optional): If set, strip any inline funcs from string
                 as if they were not there.
+            return_str (bool, optional): If set (default), always convert the
+                parse result to a string, otherwise return the result of the
+                latest called inlinefunc (if called separately).
             **reserved_kwargs: If given, these are guaranteed to _always_ pass
                 as part of each parsed callable's **kwargs. These  override
                 same-named default options given in `__init__` as well as any
@@ -246,7 +254,8 @@ class FuncParser:
                 callable (like the current Session object for inlinefuncs).
 
         Returns:
-            str: The parsed string, or the same string on error (if `raise_errors` is `False`)
+            str or any: The parsed string, or the same string on error (if
+                `raise_errors` is `False`). This is always a string
 
         Raises:
             ParsingError: If a problem is encountered and `raise_errors` is True.
@@ -295,6 +304,7 @@ class FuncParser:
 
                 if curr_func:
                     # we are starting a nested funcdef
+                    return_str = True
                     if len(callstack) > _MAX_NESTING:
                         # stack full - ignore this function
                         if raise_errors:
@@ -328,6 +338,8 @@ class FuncParser:
             if not curr_func:
                 # a normal piece of string
                 fullstr += char
+                # this must always be a string
+                return_str = True
                 continue
 
             # in a function def (can be nested)
@@ -470,7 +482,8 @@ class FuncParser:
                         # exec_return should always be converted to a string.
                         curr_func = None
                         fullstr += str(exec_return)
-                        exec_return = ''
+                        if return_str:
+                            exec_return = ''
                         infuncstr = ''
                 continue
 
@@ -484,6 +497,61 @@ class FuncParser:
             for _ in range(len(callstack)):
                 infuncstr = str(callstack.pop()) + infuncstr
 
+        if not return_str and exec_return != '':
+            # return explicit return
+            return exec_return
+
         # add the last bit to the finished string and return
         fullstr += infuncstr
         return fullstr
+
+    def parse_to_any(self, string, raise_errors=False, **reserved_kwargs):
+        """
+        This parses a string and if the string only contains a "$func(...)",
+        the return will be the return value of that function, even if it's not
+        a string. If mixed in with other strings, the result will still always
+        be a string.
+
+        Args:
+            string (str): The string to parse.
+            raise_errors (bool, optional): If unset, leave a failing (or
+                unrecognized) inline function as unparsed in the string. If set,
+                raise an ParsingError.
+            **reserved_kwargs: If given, these are guaranteed to _always_ pass
+                as part of each parsed callable's **kwargs. These  override
+                same-named default options given in `__init__` as well as any
+                same-named kwarg given in the string function. This is because
+                it is often used by Evennia to pass necessary kwargs into each
+                callable (like the current Session object for inlinefuncs).
+
+        Returns:
+            any: The return from the callable. Or string if the callable is not
+                given alone in the string.
+
+        Raises:
+            ParsingError: If a problem is encountered and `raise_errors` is True.
+
+        Notes:
+            This is a convenience wrapper for `self.parse(..., return_str=False)` which
+            accomplishes the same thing.
+
+        Examples:
+            ::
+
+                from ast import literal_eval
+                from evennia.utils.funcparser import FuncParser
+
+
+                def ret1(*args, **kwargs):
+                    return 1
+
+                parser = FuncParser({"lit": lit})
+
+                assert parser.parse_to_any("$ret1()" == 1
+                assert parser.parse_to_any("$ret1() and text" == '1 and text'
+
+        """
+        return self.parse(string, raise_errors=False, escape=False, strip=False,
+                          return_str=False, **reserved_kwargs)
+
+

evennia/utils/tests/test_funcparser.py

@@ -205,6 +205,40 @@ class TestFuncParser(TestCase):
         ret = self.parser.parse(string, escape=True)
         self.assertEqual("Test \$foo(a) and \$bar() and \$rep(c) things", ret)
 
+    def test_parse_lit(self):
+        """
+        Get non-strings back from parsing.
+
+        """
+        string = "$lit(123)"
+
+        # normal parse
+        ret = self.parser.parse(string)
+        self.assertEqual('123', ret)
+        self.assertTrue(isinstance(ret, str))
+
+        # parse lit
+        ret = self.parser.parse_to_any(string)
+        self.assertEqual(123, ret)
+        self.assertTrue(isinstance(ret, int))
+
+        ret = self.parser.parse_to_any("$lit([1,2,3,4])")
+        self.assertEqual([1, 2, 3, 4], ret)
+        self.assertTrue(isinstance(ret, list))
+
+        ret = self.parser.parse_to_any("$lit('')")
+        self.assertEqual("", ret)
+        self.assertTrue(isinstance(ret, str))
+
+        # mixing a literal with other chars always make a string
+        ret = self.parser.parse_to_any(string + "aa")
+        self.assertEqual('123aa', ret)
+        self.assertTrue(isinstance(ret, str))
+
+        ret = self.parser.parse_to_any("test")
+        self.assertEqual('test', ret)
+        self.assertTrue(isinstance(ret, str))
+
     def test_kwargs_overrides(self):
         """
         Test so default kwargs are added and overridden properly